openfun / arnold Goto Github PK
View Code? Open in Web Editor NEW:construction_worker_woman: Deploy your applications to Kubernetes with Ansible
License: MIT License
:construction_worker_woman: Deploy your applications to Kubernetes with Ansible
License: MIT License
OpenShift ressources (builds, deployements, pods, jobs) must be purged.
We can do it automatically with some configurations on templates.
The preprod
environment name is not coherent with what we use in our apps.
Rename the preprod
environment to preproduction
Current hosts pattern should be improved as:
For feature deployment, move the original feature route:
lms_host: "{{ project_name }}-lms--{{ feature_title }}.{{ domain_name }}"
to something like:
lms_host: "{{ feature_title }}.lms.{{ project_name }}.{{ domain_name }}"
And for common routes:
lms_host: "lms.{{ project_name }}.{{ domain_name }}"
lms_host: "previous.lms.{{ project_name }}.{{ domain_name }}"
feature_title
, and I think it's a great idea since the deployment will be generated by the CI platform.project_name
should be {{ env_type }}.{{ customer }}
instead of {{ env_type }}-{{ customer }}
Expected behavior/code
Memcached should be deployed in "Replace" mode (see the Redis app) which implies having 2 instances mounted on the same volume just before switching from the blue to the green stack. This may screw-up its state,
"Replace" mode deployments imply a (very short) downtime. Memcached should therefore be in a separate app so that we can avoid these downtimes as much as possible.
Actual Behavior
Memcached is included in edxapp and is therefore:
Proposal
We propose moving Memcached to its own app. It will also allow sharing it between applications...
The following warning message appeared recently:
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.23) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
It should be fixed!
OpenShift can group services belonging to the same app in the console. While the monitoring a project (customer + environment), we want to see all pods of one app, this can be achieved thanks to the app name filter.
{{ apps.name }}
in templates to group all services under the same app in OpenShiftAdd to the CI :
We project to be customizable so that anyone can use Anrold as a tool to build their own project.
Add a new directory with the following structure:
├── examples
│ └── hello
│ └── templates
│ └── ...
│ └── group_vars
│ └── ...
│ └── openedx
│ └── templates
│ └── ...
│ └── group_vars
│ └── ...
│ └── richie
│ └── templates
│ └── ...
│ └── group_vars
│ └── ...
In order to deploy one of these projects, the scripts should run docker mounting the templates
and group_vars
directories in the *arnold directory.
files
directory as it won't be used for the moment_docker_run
bash command to take the project name as an argument.We use OpenShift's initContainers
to generate custom settings for an edxapp environment. This definition is repeated among multiple templates (DCs and jobs). This is not optimal. Even if we like DC or job templates to be fully declarative and self-contained, there is room for improvement.
My first attempt was driven by Jinja's template inheritance, but it was a dead-end as the Ansible template
lookup was failing when using the {% include "_partials/initContainers.yml.j2" %}
template tag. I think it's the way to go but I did not succeed and postponed it to later.
This issue is a follow-up of the issue #33 ; we need to improve code testing.
Add to the CI :
Add possibility to plug datadog APM to the apps.
We used the $ANSIBLE_VAULT_PASS
variable to avoid having to type encrypted files ansible-vault
password. This is no longer the case. I prefer having to type this password once in a while when creating OpenShift secrets.
Remove ANSIBLE_VAULT_PASS
from env.d/*
When a deployment is finished we have to go to OpenShift to see the routes.
Print the route urls as a debug message in Ansible at the end of the deployment process.
This issue is a follow-up of the issue #4 ; we need to test the deployment strategy.
minishift
locallyinit_project.yml
playbook with the hello projectdeploy.yml
playbook with the hello projectOn each new project the Dev or Ops must do :
To bootstrap a project with all apps on the current route, we can create a bootstrap_project.yml
playbook who does these 3 steps.
The OpenShift host is set in the env.d environment files. This means that we could end-up mistakenly deploying a "feature" environment to the production host for example.
Set the K8S_AUTH_HOST
in group_vars and jinja templates instead of env.d
files.
openshift_host
in group_vars:K8S_AUTH_HOST
environment variablecreate_objects
playbook to enforce the host
for all tasks in the playbook as specified in openshift_raw
documentation https://docs.ansible.com/ansible/2.5/modules/openshift_raw_module.htmlRoutes must be created from the first application deployment, not during initialization and then patched during the deployment.
create_static_services_routes.yml
template as it's almost a copy of the deploy_patch_route.yml
templatedeploy_get_stamp_from_route.yml
tasks work without any route already created for the projectCurrently, the creation of static routes is generic and based on the presence of host variables in a service.
If we use the directory apps auto-discovery method, we do not have this host variable on dict.
On app default var file (main.yml) create list whit the services who need's route.
The generic patern of the routes is : (previous|current|next).{{ app.service.name }}.{{ project_name }}.{{ domain_name }}
For more accessibility can we rename the index.md files to readme.md ?
OpenShift labels are a useful way to filter objects and act on it quickly in the console or via the oc
CLI. When using the console, objects are restricted to a namespace (customer + env_type), so there was no real need to add the customer or env_type label. Now, that we are using Datadog to monitor OpenShift, adding those labels would be useful as they are used a meta-data to qualify objects (pods, etc.) in Datadog's interface.
customer
and env_type
labels to every objectsThe bin/bootstrap command starts by deleting any existing project. It does not ask for confirmation so there is a risk of someone unintentionally losing information.
Currently, our deployments create new complete stacks including routes to access each stack.
In a functional architecture, we need to manage routes differently to allow uninterrupted service when making a new deployment.
Stop creating routes together with the stack of svc/dc/ep.
Create only 3 routes for each service. The following hosts are examples for the lms:
Each time we make a new deployment, the following actions are taken:
Each time we confirm a deployment by triggering a route switch, the following actions are taken:
We currently only support one host per service definition to create underlying routes. As some apps (like edxapp
) may have many routes pointing to the same service (e.g. nginx), we need to improve hosts management.
Move the app.service.host
type from a simple string to a list of dictionaries with the route, the targeted service name and the targeted port.
An env.d/development
file is required to start using Arnold with Minishift.
We want the default environment to be used without any work from the developer:
env.d/base
to env.d/development
.As mentioned in #76, Arnold's docker image is a tool, so it can be ubuntu-based instead of debian-based. Switching to Ubuntu would allow to use more recent packages, particularly, python 3.6.
ubuntu:18.04
base imageWe need to automate the user account creation for every Django service/flavor/customer Arnold will deploy.
Add a create_users
job, that will actually create a superuser and other profiles using django management commands and vaulted credentials.
We wrote bash scripts to ease the developer experience. For now, they are intended to get used only for development purpose. But, I think we must also consider using those scripts in other environments (even staging or production) to manually interact with a running stack.
I see two main advantages in doing so:
We need to make environments more configurable instead of hard-coding minishift-specific rules.
.vault_pass.sh
was used to ease ansible vault decryption from OpenShift. It is no longer required.
Remove this file from the repository 😄
Expected behavior/code
A symbolic link app in apps dir must be included in available_apps
Actual Behavior
A symbolic link app in apps is ignored.
Environment
KVM is presented as the preferred option to run Minishift. Most developpers will already have VirtualBox on their laptop but not KVM. Furthermore we experienced network issues when running KVM that made us prefer VirtualBox.
Replace KVM by VirtualBox:
Some objects are created in environments where they should not e.g. endpoints in development/feature, acme in development/feature, database containers in staging/preproduction/production, etc.
Improve the structure of objects in group_vars to allow this configuration:
Historically, this project has been initiated using our own GitLab instance, hence, we designed a GitLab-CI-based continuous integration (CI) workflow. When we decided to open-source this project and move it to GitHub, we lost our CI. As it is a prerequisite for this project (for every project in fact), we need to improve and re-implement it.
As we usually use CircleCI, let's stick to it for Arnold. We have the advantage that CircleCI and GitLab-CI philosophies are quite close. Therefor, we might thought that this will be an easy task, but it's not. The way we designed our original CI workflow is not sustainable for an open source project (the last step is a deployment to our OpenShift instance).
My first idea is to use a trashable MiniShift instance in our CI workflow, but this will be a really tricky part.
Dear contributors, I invite you to make proposals; nothing has been decided yet.
We used to deploy to production in our default CI. This is no longer the case.
Remove the following vaulted secret: group_vars/secret/patient0/production/richie/credentials.vault.yml
As mentioned in #89, at the time of writing, our CI fails at the test-bootstrap
job that raises a timeout error from CircleCI (jobs cannot take longer than 10 minutes to execute).
To bypass CircleCI limitation, we will try to split the test-bootstrap
job in multiple jobs (ideally one per app). 🤞
Adding a customer to deploy may require creating configMaps, vaults, etc. It may be useful to automate this task.
Create a add_customer.yml
playbook that:
group_vars/customer/foo/main.yml
from a template file (customer_main.dist.yml
)group_vars/customer/foo/{{ env_type }}/main.yml
from the same template file (customer_main.dist.yml
)group_vars/customer/foo/{{ env_type }}/secrets/
directorygroup_vars/customer/foo/{{ env_type }}/configs/
directorywith foo
the name of this new customer.
The playbook that creates configmaps automatically find them in the templates/configmap
directory and pushes them all to OpenShift. We want to be able to define which configmaps are created for each customer or env_type.
For example, the "edxapp-fixtures" configmap should only be created for the development
or feature
environments.
Apply the same mechanism as for OpenShift objects to configmaps:
openshift_configmaps
that can be overridden by customer/env_type,openshift_configmaps
variable.OpenShift jobs to run migrations or collectstatic tasks for deployed apps are currently unreliable since they are created as is even if the targeted service is not up yet. We need to be able to:
oc
to get the job execution status (wait loop)vars/main.yml
configWe now have too many playbooks at the project root.
playbooks/
directoryUsing minishift
and thus a VM to run OpenShift adds a complexity layer for development and is not required. Let's switch to an all-in-one docker container that runs OpenShift!
oc cluster [up|down]
* /etc/docker/daemon.json
{
"insecure-registries": [
"172.30.0.0/16",
]
}
Routes defined in group_vars/all/main.yml
are overridden by group_vars/all/openshift_routes.yml
. At the time of writing the cms & lms routes have been inactivated and are thus not created by the create_objects
playbook / task list.
Un-comment the following two lines and we will be good to go:
https://github.com/openfun/arnold/blob/master/group_vars/all/openshift_routes.yml#L9
The files/
directory has been left almost empty, and its purpose may not appear obvious as it is not documented at all.
The files/
directory may be used to override project's configMap
(see tasks/create_config.yml
). Let's document this feature!
Arnold's image takes a long time to build. It should not be a necessary step for newcomers.
The Continuous Integration in CircleCI should push Arnold's Docker image to Docker Hub after building it.
This has already been done on Richie here: https://github.com/openfun/richie/blob/master/.circleci/config.yml#L432
We need to configure health checks.
I propose following Mozilla's Docker flow:
When a route in not defined OpenShift serve the default 404 page : Application is not available
Create a 404 template to serve a 404 page.
Use Ansible possibility to :
To have a exemple of functional CI on Gitlab. It should not be usefull to have a gitlab-ci in parallel of Circle CI.
As a first approach, we drafted a project documentation as a navigable suite of markdown formatted files. Let's switch to a more convivial documentation pushed to readthedoc.
Constraints
Implementation
In the development environment, we want to have more tools installed on our images.
Use dev Docker images only for the development
environment:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.