eimis-ans / eimis-synapse

Make a Synapse homeserver work on a managed kubernetes server hosted by OVH

Synapse on Kubernetes

Runs a Matrix server on a managed kubernetes cluster hosted by OVH.

Features

• Customized Synapse server as the Matrix messaging server along with it's database and s3 bucket
• Customized Keycloak as an alternative ID provider along with it's database
• Element-Web as a Matrix client
• Prometheus et Grafana for the monitoring
• The stack is also configured in a specific and configurable way cf. the ansible part.

Prerequisites

• an account in OVH hosting provider and its credentials (application key, application secret, consumer secret and endpoint)
• to store Terraform state files : a S3 object storage with the credentials to connect to (access key, secret key, endpoint and region) and a bucket named terraform-states-hp-myenv for example.
• a user and credentials dedicated to openstack with the following rights : [Network Security Operator, Volume Operator, Network Operator, Backup Operator, Compute Operator, Image Operator, Administrator, Infrastructure Supervisor]
• to reach the future Synapse homeserver : a valid dns zone hosted by OVH
• to send some mails to users : a valid access to a SMTP service

On the linux running this code :

• the terraform CLI
• the ansible tool
• the kubectl tool
• additional needed packages : openssl, yq

Provisioning infra

The following steps will setup various OVH resources necessary to run the Synapse homeserver.

Tip

The Octavia load balancer is only useful if you want the cluster to be isolated from the web. If it's not necessary don't use it and remove type: NodePort in ansible/roles/ingress-controller/tasks/templates/ingress-nginx-service.yml and manually set the dns entry

---

Tip

The other thing you may want to consider if not running an official production environment is to get rid of admin_vm this part was done to make sure the alert manager was able to reach an hypothetic Zabbix server on the admin private network.

• Create in the local folder a local.env.sh file copying the script/local.env.template.sh file and fill it with all the environment variables values needed. OS_variables relate to the openstack part.

  Then source this file :

  source local/local.env.sh

• Generate the var file for provisioning stage (terraform.tfvars) based on values previously set :

  sh scripts/generate_provisioning_var_files.sh

• Go to the terraform folder

  cd terraform

• Initialize the Terraform workspace specifying the name of the S3 bucket

  terraform init -backend-config="bucket=terraform-states-hp-$ENVIRONMENT"

• Create the Terraform execution plan to validate that everything is ok

  terraform plan

• Apply the Terraform plan

  terraform apply

  This will lead to the creation of a kubernetes cluster with 1 control plane node and several worker nodes

Configuration

The configuration part will be done with Ansible and is quite independent from the provisioning part.

• Generate the files (kubeconfig-$ENVIRONMENT.yml, ansible/group_vars/all.yml) and vars needed :

  ./scripts/generate_configuration_var_files.sh

  For more info on kubeconfig file see https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/

• And then execute :

  ./scripts/ansible_configuration.sh

This will lead to the installation of the following components in the cluster :

• basic components :
  • an ingress controller
  • a certificate manager
• components specific to our stack :
  • a Keycloak instance along with its operator
  • the synapse stack and its customization
  • the element-web stack
  • the stunner stack used to facilitate audio/video on element
  • a prometheus/grafana stack for monitoring
  • an alpha unofficial version of a MS teams bridge

Other credits

• The Matrix-synapse stack is based on the work done by Alexander Olofsson : https://gitlab.com/ananace/charts/-/tree/master/charts/matrix-synapse
• Technology icons created by juicy_fish - Flaticon

eimis-synapse's Issues

Update of users in directory

How to update users in the directory :

• if the update comes from the RPPS : synchronization ?
• if the update comes from other directories ?

Better way to ensure persistence of data between re-installations/updates

During the release of sprint 1, we had to delete the entire stack and recreate it in order to avoid problems with persistentVolumeClaim matrix-synapse created on a certain node and not transferable on one another.

It would be interesting to externalize :

• data storage on postgresql out of kubernetes
• other file storage on other external solutions (s3 ? media-repo ?)

In order to keep this data safe between 2 installations/upgrades

takslist :

• postgresql database managed by OVH created via terraform
• private network created via terraform
• connection between database and Kubenertes for synapse deployment
• cleanup of deprecated statefulset
• data migration to new postgresql
• restore data test

Definition of done :

• data transfered between K8s and new database
• external postgresql database
• private network between k8s and database
• restore data test done

Connecting non brigded IM : Pandalab case

What about connecting Matrix with IM that don't have a standard bridge ?
Pandalab has develop a Javascript based bridge and plan to operate its own matrix server to connect the bridge to.

Tasklist :

• Find a way to collaborate with Pandalab around the bridge they are building

Defintion of done :
collaboration established

Prepare the recruitment of a additional tech profile

Follow up - Activate the ability to have HDS on our OVH account

To activate the HDS option on OVH public cloud there are some prerequisites :
https://www.ovhcloud.com/fr/enterprise/certification-conformity/hds/
It can be activated in OVH UI in project settings :

But there is a prerequisite to fulfill before getting HDS option available : the project must have a support level set to "Business" or "Entreprise" (cf. https://www.ovhcloud.com/fr/enterprise/certification-conformity/hds/) and it will cost :

• Business (https://www.ovhcloud.com/fr/support-levels/business/) : 10 % of the monthly bill with a minimum threshold of 250 € HT/month (without VAT)
• Entreprise (https://www.ovhcloud.com/fr/support-levels/enterprise/) : we must get in touch with customer service to get informations ...

Definition of done :
HDS mode can be activated on OVH UI.

Search mechanism : customization (see with tchap) ?

Astonishment report about the enrolement and authentication workflow of PSC

Definition of done :
Report written and sent to ANS

Synapse on kubernetes : other installation methods ?

Until now we are using the following helm chart to install synapse on kubernetes :
https://gitlab.com/ananace/charts/-/tree/master/charts/matrix-synapse

Other solutions exists :

• a fork of this chart, made by gematik : https://gitlab.opencode.de/bwi/bundesmessenger/backend/helm-chart
• another chart made by Matthias Goerens : https://github.com/mgoerens/matrix-chart
• an operator (alpha stade) made by Matthias Goerens : https://operatorhub.io/operator/synapse-operator / https://github.com/opdev/synapse-operator
• the matrix operator used by element team : https://github.com/vector-im/ess-starter-edition-core

It may be interesting to study this other solutions and see if they are more relevant than the one we use.
What about the scaling of a worker ?

Give a try to Fluffy Chat

https://matrix.org/ecosystem/clients/
To test : Fluffy Chat.

Task list:

• list differences
• test web and android and apple
• does it make sense to test this differences on end users

Definition of done:

Produce a small report compiling these results

Prepare presentation material for ANAP workshop

Expose Synapse server to outside world

As a user,
I want to connect my Element app to the synapse backend
So that I can send a message on the chat

Make the synapse reachable from outside the cluster

• get a domain name to reach the cluster
• create an ingress-controller in K8S cluster
• configure the ingress-controller to manage the matrix ingresses
• configure a DNS zone to point to the loadbalancer IP previsouly created

Connect an Element client to synapse:

• Ask help to understand which endpoint must be reached by element and how it works
• refine the synapse configuration to allow the connection

Prosanté Connect - real connexion

Connect our matrix instance with ProSanté Connect.

Tasklist:

• fullfil the eimies registration on ANS portals
• get technical documentation and accesses
• modify eimis instance to connect to PSC

Definition of done :
A user identified with Prosanté Connect can access the eimis matrix instance

Get in touch with BA and BC to know if they should join the team

Bridge mecanism analyze

Different types of bridges are available :

• https://matrix.org/bridges/
• https://matrix.org/docs/guides/types-of-bridging

Which one should be used by which Health Instant Messenger editor ?

Documentation about the configuration needed for private federation to work

gather the configuration modifications needed to let the public and private discussions happen on eimis instance between users from external home servers

Connexion through CIBA Flow on mobile phone

EIMIS with its own Github organization

Tasklist :

• open eimis organisation on Github
• migrate repo to this organisation
• invite users : Juliette, Julien , Raphaël, heroinedor
• apply best practices from beta gouv
• update documentation (wiki) and send mail to partners to update URL

Authentication with ProSanté Connect - analyze

Healthcare Professional can connect to some IM with an authentication tool developed by ANS : proSanté Connect.
How will EIMIS be related with PSC ?
Should it be also connected to PSC?

Sub-tasks :

• get technical and functional documentation about PSC
• get an access to a PSC development environment (URL, login, process, ...)
• Understand link with RPPS

Mapping of properties between PSC data and synapse user properties

Close registration on synapse

The first installation of synapse was made with no control on the registration in order to set it up quickly.
But as said in the Matrix documentation (https://matrix.org/docs/guides/understanding-synapse-hosting#not-leaving-the-door-open), it is now time to "close the door" in order to avoid some problems.

Private federation of matrix homeservers

The targeted product will need to have a private federation of homeservers operated by editors and eimis team.
Tasklist to achieve this goal :

• URL list of editors homeservers
• find a way to facilitate the onboarding of a new editor in the federation : a form must be filled by the editor to integrate federation. This form can be : github pull request, github issue, standard excel form, google form ....

Federate dev and preprod instance

To simulate multiple client instances, we might need to setup a federation between our own homeservers by updating homeservers and reverse proxies configurations.

Connexion of preprod to ProSanté Connect

Now that develop instance is connected to PSC, we should connect the preprod instance.

Tasklist :

• see with PSC team to have another connexion declared in PSC
• find a way to have the correct picture linked to the connexion button (PSC prerequisites)

Defintion of done:
we can connect to preprod instance through PSC

private room : discussion between several users from distinct apps

as a user with MXID from A messenger
I want to open a private room with user of B messenger and send messages to him
so that I can dicuss privately with him

Definition of done :
screenshot of the discussion

eimis public room : discussion between 2 users from distinct apps

as a user with MXID from B messenger
I want to enter the public room of eimis instance and see messages from user of A messenger
so that I can answer to him

Definition of done :
screenshot of the discussion

Manual deploy with CI without pushing code

Before each week-end the develop infrastructures are destroyed with destroy workflow on CI in order to save money on OVH account.
So each monday the deploy workflow must be executed on CI to redeploy the infrastructure.
But the deploy workflow can only be triggered through a code push.
So this workflow must be modified in order to trigger it manually

Teams bridge analyze : find open source alternative

As an eimis user,
I want to be able to chat with a health professional that as an account on teams.
In order to begin a chat with him

This implies to study the Teams bridges :

• SaaS closed version : https://element.io/blog/ems-launches-bridging-for-microsoft-teams/ implies to get in touch with Element.
• Open source versions :
  • old abandoned version : https://github.com/neilsb/mx-puppet-teams
  • other options : turt2live/matrix-wishlist#16 ?

Tasks :

• investigate to find all open-source possibilities
• analyse each possibility
• write synthesis

Definition of done :
Synthesis of investigation.

End-to-end data encryption

What about the end-to-end data encryption when 2 non-matrix IM are talking together ?

Is it possible to have encryption at the bridge level ?
Or is it mandatory to use a separate tool ?

CGU

cf. solution proposed by Antoine : https://www.sesali.fr/ncpehfr-gui/index.html

Architecture decision for directory : federation, centralization, hybrid ?

Problem to solve : federation vs centralization of the directory :

1. Option A - centralization :
   1 centralized directory where ANS stores all the healthcare professionals and other professionals.
   Each user in the directory has 1 MXID = 1 n° RPPS, except for non healthcare professional

2. Option B - federation :
   Every ANS partner has its own directory of healthcare professional and they are all federated.
   How to deal with different MXID for one RPPS n° ?

3. Option C - Hybrid :
   ANS manages the centralized directory dedicated to healthcare professionals
   Partners manage the directories dedicated to non healthcare professionals
   All these directories are federated .
   1 MXID = 1 N° RPPS for heathcare professionals
   potentially several MXID for each non healthcare professional

Ideas :

• organize a hackathon / chaos ingeneering workshop to see how these hypothesis react with different system failures (central server of ANS, partners editors, ...)
• Best options for users seems to be A and C. We could first test option A and then complete it with option C

Directory federation : which other directories to federate with RPPS ?

private room : 1-1 dicussion between 2 users from distinct apps

As a user with MXID from A messenger
I want to open a private room with a user from eimis instance and send messages in it
So that I can discuss privately with this user

Fédération

Add Element web to stack

Ajout du client web Element pour permettre aux partenaires de tester sans quitter le domaine eimis.incubateur.net

Directory search functionality

How to implement the search functionality in the directory from the search UI in matrix ?

Directory federation : technical architecture

Maybe based on a keycloak instance ?
RPPS main and keycloak secondary ?

Add mention of KUBECONFIG var in documentation

To execute kubectl on local machine, the KUBECONFIG variable must be set.
So it must be added to the README.md documentation

Plug synapse to keycloak to simulate Prosanté Connect

Since ProSanté Connect is compatible with OIDC (openid connect), it can be emulated with keycloak.
This is only a way to wait for a real access to PSC.

Invite editors to apply to a synchronized sprint

Manual fulfillment with extract from RPPS and partners directory

Quick and dirty: Explore discovery room solution for user directory aggregation - eimis federation

As a user from eimis preprod,
I want to find another user from eimis develop with whom I have never discussed before,
In order to start a conversation with him.

to allow users from different IM apps to retrieve each other and being able to send private messages across homeservers, we can use the discovery room principle conceived by the Tchap team.

Prerequisites :

• At least two federated homeservers (dev and preprod will be fine)
• A hidden discovery room on every homeserver
• A dummy user per homeserver
• Join all users to their respective homeservers

Documentation : https://www.notion.so/Discovery-Rooms-77c646f7e348436a989c812ea39712f9

Definition of done :
A user from develop instance can be found from preprod instance and inversely.

Bridge Whatsapp ?

Following a discussion with an editor it has been raised the issue of the onboarding.
Since most of the current Healthcare Professionals are using Whatsapp to communicate, maybe it could be a relevant option to set a whatsapp bridge in order to make easier the switch from Whatsapp to the IM?

OVH payments

Solve the OVH payment problem : find a process that fit with ANS needs.
Ask help to Julien D.

Conditions to fulfill to have HDS provider

standard format to exchange health data through IM

Some IM editors provide to healthcar professional the ability to exchange data other than text and attachments through IM.
The format used is based on json.
Would it be useful to set a standard for these data, based for example on structured data used in semantic web or CDA (clinical document architecture), in order to make it possible to exchange data between IM ?

Model already existing : https://esante.gouv.fr/produits-services/mos-nos ?

RGPD

eimis public room : dicussion with one user from external homeserver

As a user with MXID from A messenger
I want to enter the public room of eimis instance and send messages in it
So that I can discuss with users from eimis instance

Feeding of the directory

Import or synchronization of users with RPPS / RPPS + ?

• If synchronization then RPPS/RPPS+ is main and our directory is secondary.
• If import then : one shot first (manual?) import or regular automatic imports ?

Start the reflexion about the experimentation legal frame, on the Dinum end

ANS RSSI