sci-oer / automated-builder

An automated build tool to generate custom sci-oer docker images

Automated SCI-OER Image Builder

This is an automated build tool to generate customized versions of the language specific versions of the base-resources images.

Typically, a customized docker image can be created by extending a Dockerfile with the new modifications and setup script and your good to go. However, some of the tools that are prepackaged in this container can not be customized and configured without the services running (wiki-js). This service aims to solve this problem by running a version of the base image, customizing it, extracting the configuration files, then repackaging them into a new Docker image.

Requirements

In order to run this project you must have the following installed and running on your machine:

• docker
• git
• python3

The build script will start a docker container in the background, so it can be customized then build a new docker image. The user that is running the build script must have permissions to run docker commands.

Git must be installed in-order to load and configure the seeded content.

Building this Docker Image

This image has been published to Docker Hub, but the follow command can be used to build the image locally.

docker build \
    --build-arg GIT_COMMIT=$(git rev-parse -q --verify HEAD) \
    --build-arg BUILD_DATE=$(date -u +"%Y-%m-%dT%H:%M:%SZ") \
    -t scioer/automated-builder:latest .

Creating a Custom Image

This script will create a sci-oer image that is customized with the following:

• a set of pre-made lecture content
• one or more example project repositories.
• wiki content to be loaded from a git repository
• a set of Jupyter lab notebooks
• a customized wiki title

There are a number of configuration options that can be specified when building the customized container. Run scioer-builder --help to see a complete list of configuration options and how they can be used.

All the options are optional, if one is left out then no content will be configured.

There are currently three methods that can be used to create a customized image.

1. Using the published scioer-builder package (preferred)
2. Using a prebuilt docker image
3. Calling the scioer_builder/cli.py script directly

Using Docker

For convenience a prebuilt Docker image has been published that includes all the dependencies needed to generate a custom image.

For the automated-builder container to be able to create the custom docker image it must have the docker socket passed into the container.

$ docker run -v /var/run/docker.sock:/var/run/docker.sock \
    scioer/automated-builder:latest <options here>

NOTE: If you are using the --key-file option when running with docker, the path that is specified must be the path in the container, not on the host machine

Using the pip package

The auto builder script has been distributed to pypi for easy use. It can be installed by running:

pip install scioer-builder

Then run as a normal command:

scioer-builder --help

Getting Help

If you need help getting the builder script to work or have questions or find any issues you can open a GitHub Issue.

Getting Involved

If you notice something that doesn't quite work or if you want to add new features or support the implementation any contributions are welcome.

Currently, we are focusing on completing this script so that all of the aspects of the sci-oer container can be customized using the cli, or possibly a configuration file. The next steps will be to develop a web interface so that the customized version can be created without needing to install all the tools.

See the CONTRIBUTING.md guide for more information.

License

This project has been published under the AGPL-3.0 License. This essentially means that you can use this for whatever you want as long as any modifications you make are published under the same license.

automated-builder's Issues

Load git ssh key directly from a specified key

Allow the ssh key to be passed in directly as a string using the --ssh-key flag.

This flag is probably not the best option to load the the key as it will be visible through the command history.

add workflow to automatically create a GH release

This should create a GH release very time a version tag is pushed to the repository.

This should include the build artifacts as well as the changelog.

Automatically upload the image to a specified registry

Related: #43

Is your feature request related to a problem? Please describe.

Generating the docker image is only useful if it is in a form that can be accessed by students.
Uploading the image to a docker registry will make it easier for students to download the image.

Describe the solution you'd like

A set of cli flags can be used to specify the registry that the image gets pushed too.

--registry, --registry-user, --registry-password, --registry-password-file.

Describe alternatives you've considered

An alternative implementation is described in #43, both should be implemented.

Add labels to the main docker file

There are labels present for the oo-resources custom Dockerfile but not for the automated builder file

Fill in the README

The readme file currently only contains the default content that github put in the file.

• compilation instructions
• installation instructions
• how the application can and can not be used (it must be run from within a docker container at the moment and needs to have access to the docker daemon that is running the the image.

add workflow to deploy pip package

The pip package should be kept up to date by deploying a new version every time a new tag is pushed with a version bump.

su from root after startup?

once the container has started, is it possible to su to the student user?

The git ssh host does not automaticly get accepted and it requires user intervention

running git clone from within a container should not require user interaction

Add cli flags to pass options into the script

argparse or docopt are some potential options.
This will allow the cli generator to be used independently of hardcoded options.

Load git ssh credentials from a specified key file

Using the --key-file flag tell the build script where to load the the ssh key used for git authentication from.

add CI pipeline linter

Since this project contains more code than the oo-resources project a linter should be added to help ensure that the code maintains readable and maintainable.

additional content

in addition to the wiki and the jupyter tutorials there is a set of practice problems that go, predictably, in the /course/practiceProblem folder. There will also be a set of video lectures that go in the /course/lectures folder

Automatically export the image as a tar archive

Is your feature request related to a problem? Please describe.

Generating the docker image is only useful if it is in a form that can be accessed by students.
By generating a tar archive that can be downloaded and distributed to students is a way to achieve that.

Describe the solution you'd like

A cli flag that can be used to specify the name of the tar archive to produce in the output folder.

Describe alternatives you've considered

I have also considered being able to automatically push the generated image to a docker registry.
This should be an easier implementation than the full docker push implementation and could be supported as well.

Additional context

There may be issues with with naming conflicts of the generated tar archive.
This could be remedied by appending a timestamp the the file name that is generated and then printing the file name so it can be used by the caller.

Add the ability to set git credentials when the seed data is pulled

Currently the git repository that is being cloned to get the seed content for jupyter lab must not require authentication.

It should support optional username + password credentials as well as ssh key credentials

create video demo of OER construction process

Load the ssh key file from an envirionment variable

Allow an environment variable like SSH_KEY_FILE to be used to specify the location of the ssh private key file instead of using the cli flag.

Automate setup of wiki seed content

Moved from sci-oer/java-resource#10

The wiki can be configured to import the posts from a git repository, however this needs to happen while the application is running.

Currently any seed content must be added manually, this is not conducive to automatically creating the image.

An image creation script should be developed that will:

1. start the base image without any customization
2. configure the storage git repository:
   • set the URL
   • set the branch
   • set the ssh private key
   • configure to pull only from the repo
3. run the import from git repo
4. remove the git repo from the storage settings
5. deactivate the git repo storage setting
6. upload logo image asset
7. set site information
   • title
   • logo file
   • organization name
8. revoke api key
9. stop the docker container
10. copy the database.
11. create new image with the seed database

Change mapped drive structure

The /course directory should have the following subdirectories:
coursework
jupyter
lectures
logs
practiceProblems
wiki

The /course/jupyter folder should put the pre-written tutorials in a /tutorials folder instead of /builtin

Add a CI pipeline to generate the docker image

Use a similar ci pipeline as the oo-resources container to build and publish the image

gradle vs gradlew

There will be at least 20 example projects included with the container, probably more. I'm concerned about the impact on container size if we include multiple sets of gradle wrapper files. thoughts?

Create Changelog

A Changelog should be created for this project that gets updated every time a feature is added or a change is made

Will not use git credentials for private repos

Describe the bug

the ssh key is not used for any repo other than the wiki

To Reproduce

Use a git repo that requires authentication for the examples

Setup inital pytest suite

Automated tests are a good way to ensure that the module works as intended and that each portion can be re-factored in the future while ensuring the behaviour is the same.

Pytest is a testing framework for python that will allow this to be accomplished in a fairly straight forward way.

student proof install/running

Need to make instructions for grabbing the container (assuming its been uploaded to the socs dockerhub) and for setting it up to be run from docker desktop. Instructions need to work on os/x as well as windows.

home directory for student user

The home directory for the student user should be /course or should have an alias to /course in it.

Figureout how to preseed the wiki content by modifying the wiki.js storage dirrectly

Currently the pre-seed wiki posts are created manually by directly interacting with the graphql endpoint.
This will not permit the upload of files, only text posts. This is not sufficient for use in this educational resources.

A better approach would be to use the graphql api to configure the git storage backed and trigger an import. This will allow all wiki content to be loaded. However I currently do not know enough about how the wikijs api works or how graphql works to be able to implement this.

location of practice problems

The wiki links to practice problems are set up so that the path looks like:
/course/practiceProblems/dataStructures/hashMap.
The current builder makes paths that look like: /course/oo-oer-examples/practiceProblems/dataStructures/hashMap

I can rename the oo-oer-examples repo to practiceProblems if that would make it easier. Or the builder should take the contents of the repo and put it into the practiceProblems subdir.

Import lectures from local directory

As an alternative from keeping the lectures in a git repository allow the lectures to be imported from a local directory.

The method of loading the lecture content should be mutually exclusive.

Set and document the ssh credential heiarchy.

In addition to allowing the ssh key to be loaded using various methods a set of prescience rule should be set and documented to state which will be used if multiple are specified.

Possible ordering, the highest number takes priority:

1. a potential config file
2. SSH_KEY envirionment variable
3. SSH_KEY_FILE environment variable
4. --ssh-key or --key-file cli flags, mutually exclusive

add workflow to automatically increase the semantic version number

Currently there are 3 spots where the version number must be updated for each release.

• version.py
• Dockerfile
• setup.cfg

It would be easy to forget the version bump in one of these locations. The workflow should take care of automatically creating and bumping the version number whenever it is needed.

Some possible options:

• every merge into main will create a release candidate build with the next version number
• tags will be made manually, the workflow will update the files on main, create a new commit, and move the tag, and create the release

Document OER construction process

Documentation must be usable by instructors unfamiliar with docker.

migration away from sqlite

newer versions of wikijs don't use sqlite. We will need to sort that out. It might be possible to migrate to a sql server and preload the wiki using sql?

Add preseed content for the worked examples

In the INTED paper we mentioned that some of the course content would only be accessible directly from the file system. We do not currently have any way to load this kind of seeded content into the image.

configure and build docker container

for your first container please use the flag that keeps the git repos intact so that you can work in the repos and push back to the main repo on gitlab easily.

Fill in the information for the community profile

• Description
• Code of conduct
• Contributing guide
• Issue template
• PR template
• enable code scanning
• enable dependabot

Use graphql api to configure the wiki title

Wiki customization is a nice to have for individualized course content.

• title
• locale
• favicon
• login
• admin username / password for non english users

Load configuration from a yaml file

Is your feature request related to a problem? Please describe.

Having to pass all the configuration through as cli flags or envirionment variables can be frustrating if the same configuration is used for a number of different course configurations.

Describe the solution you'd like

This should load the configuration parameters from configuration files.

A hierarchy (higher takes priority):

1. ~/.scioer.yml
2. ~/.config/scioer/config.yml
3. $XDG_CONFIG_HOME/scioer/config.yml
4. /etc/scioer/config.yml

A specific configuration file can be specified with the --config <file> or ignored with the --no-config options.

Describe alternatives you've considered

This should tie into the configuration order of precedence with #43, #44, and #45.

Add docker build instructions

Is your feature request related to a problem? Please describe.

Readme is missing the instructions to build the autobuilder image

Create pip package for the builder

Is your feature request related to a problem? Please describe.

Allow the builder script to be installed as a normal pip package

Describe the solution you'd like

Installation of the sci-automoated builder should be installable by running pip3 install sci-builder

Load configuration options from envirionment variables

Is your feature request related to a problem? Please describe.

Some of the configuration options that can be passed through cli flags should also be configured to be passable through envirionment variables.

Describe the solution you'd like

Optionally load the configuration values from envirionment variables.
The envirionment variables can also be loaded from a .env file.

Describe alternatives you've considered

This is one of several methods that is being proposed to configure the builder

Pass ssh private key as volume mount to sci-oer container

Is your feature request related to a problem? Please describe.

When cloning the git repository for the wiki mount the ssh private key in the container so that it does not need to be sent in the api call.

Describe the solution you'd like

This should mount the key in read only mode from the host into the initialization container.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update actions/cache action to v4
• chore(deps): update actions/setup-python action to v5
• chore(deps): update dependency docker to v7
• chore(deps): update docker/build-push-action action to v6
• chore(deps): update release-drafter/release-drafter action to v6
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency gitpython to v3.1.41 [security]
• chore(deps): update dependency tqdm to v4.66.3 [security]
• chore(deps): update dependency colorlog to v6.8.2
• chore(deps): update heinrichreimer/github-changelog-generator-action action to v2.4
• chore(deps): update pypa/gh-action-pypi-publish action to v1.9.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update docker/login-action action to v3
• chore(deps): update docker/metadata-action action to v5
• chore(deps): update docker/setup-buildx-action action to v3
• chore(deps): update docker/setup-qemu-action action to v3
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

dockerfile

Dockerfile

• python 3

builder/data/Dockerfile

github-actions

.github/workflows/create-release.yml

• actions/checkout v3
• actions/cache v3
• heinrichreimer/github-changelog-generator-action v2.3
• release-drafter/release-drafter v5

.github/workflows/deployment.yml

• actions/setup-python v4
• pypa/gh-action-pypi-publish v1.8.8
• docker/metadata-action v4
• docker/setup-qemu-action v2
• docker/setup-buildx-action v2
• docker/login-action v2
• docker/build-push-action v4

.github/workflows/verification.yml

• actions/checkout v3
• actions/checkout v3
• actions/setup-python v4

pep621

pyproject.toml

• setuptools >=61.0.0
• setuptools_scm >= 6.2

pip_requirements

requirements.txt

• colorlog ==6.7.0
• docker ==6.1.3
• docopt ==0.6.2
• GitPython ==3.1.32
• requests ==2.31.0
• tqdm ==4.65.0
• pyreadline3 ==3.4.1

---

• Check this box to trigger a request for Renovate to run again on this repository

Load the ssh key dirrectly from an envirionment variable

Allow the ssh private key to be specified directly using an environment variable like SSH_KEY instead of using the cli flag.

Allow the builder to be run on the docker host

Currently the automated builder must be run from within a docker container and interacts with the docker daemon on the host machine. It would be beneficial if the builder can detect when it is not running in a docker container and can change the network mode that it uses to communicate with the bare wiki image that it is customizing.

make FAQ with common issues

When getting @NLaundry, @judiMc and @CameronNorrie setup with this project a number of issues came up multiple times, having a FAQ section in the readme would make fixing those challenges easier.

these are the sections:

• https vs ssh git repos
• add vpn issue
• wierd git failure
• if it fails to connect durring the retries
• pull takes a long time
• cant pull image
• ssh key setup
• order of docker flags

docker desktop "open in browser"

if one clicks "open in browser" on docker desktop it goes to port 2222. Is it possible to change that so it goes to the wiki port?

Test if the same docker method will work in Kube

If the automated builder container is running in a kuberneties cluster will the method of interacting with the docker daemon still work or will an alternative method be needed?

The intended deployment location for this service is on the school k8s cluster.

video streaming

We need a way to stream the video lectures. Wiki.js currently doesn't support that natively.

node/knex version

So I'm trying to enable the git synchronization in wiki.js and getting the errors noted below (and it isn't synching)
a casual google points to a versioning problem maybe? https://stackoverflow.com/questions/40435315/knex-timeout-acquiring-a-connection-the-pool-is-probably-full-are-you-missing

---

2022-04-07T19:17:10.301Z [MASTER] error: Syncing with storage target git: [ FAILED ]
2022-04-07T19:17:10.302Z [MASTER] error: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
2022-04-07T19:17:29.955Z [MASTER] warn: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
2022-04-07T19:18:10.237Z [MASTER] warn: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
2022-04-07T19:19:29.821Z [MASTER] warn: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
root@2d9bb7c29bde:/course/logs#

command prompt default

the command prompt on the terminal window is not useful. Should be changed to some version of current directory.