tcibinan / flaxo

EdTech platform for practical programming courses

Flaxo

Flaxo educational platform is a pragmatic way to organise, manage and report programming courses.

Basics

The educational process using Flaxo platform can be described in five steps:

1. Tutor creates a course which is basically a simple Git repository.
2. Tutor fills the course tasks with tests that students are going to write implementations for.
3. Students solve the tasks and create pull requests.
4. Flaxo evaluates students solutions using several metrics.
5. Tutor receives an interactive dashboard with the students progress.

Features

Validations

Flaxo uses several integrated services including Travis, Codacy and MOSS to collect different metrics for all the solutions. It includes testing, code quality analysis and plagiarism analysis.

Plagiarism

All the solutions for a single course can be analysed for plagiarism. Plagiarism analysis results can be viewed in a graph viewer tool and their minified representation is listed in the course dashboard.

Generation

Flaxo can generate course\repositories with a preconfigured environment for several languages and testing frameworks. F.e. gradle project with all the required dependencies can be generated for java language and junit testing framework.

Export

Course dashboard can be exported in one of the supported formats: json, csv and tsv. It makes all kinds of integrations with the Flaxo possible.

Examples

Several University courses and a few developer trainings were hosted using Flaxo. Some of them are listed below.

• Data structures on Java Russian
• Java basics Russian
• C++ basics Russian

Documentation

Flaxo documentation is still in progress. Nevertheless some useful information on how to deploy your Flaxo instance and get started is already present. Documentation can be found here.

Contribution

Flaxo is entirely an opensource project and only the community can make a difference. So please feel free to open new issues asking questions, suggesting new features or reporting bugs.

If you intend to contribute to Flaxo project then you can found information on how to build Flaxo from sources in the corresponding documentation page.

Credits

{ 
  "full_name": "Andrey Tsibin",
  "email": "[email protected]",
  "telegram": "@Nameofthelaw",
  "vk": "https://vk.com/id24276156",
  "app_icon": "by Anton Ivanov from the Noun Project",
  "year": 2018,
  "city": "Saint-Petersburg"
}

It will be great if you add Flaxo badge to your course README.md.

[![from_flaxo with_♥](https://img.shields.io/badge/from_flaxo-with_♥-blue.svg)](https://github.com/tcibinan/flaxo)

flaxo's Issues

Validation plugins

The issue is a part of plugins system #30.

Validation plugins should be organized in a generic way which support at least the following business processes:

1. Activating a validation for a course.
2. Changing an activated validation settings.
3. Calculating validation score for each solution in range from 0 to 100 which will be used in suggesting score calculation formula.

Existing validations should be transformed to plugins supporting all the previously described business processes.

GitHub private repositories

Flaxo strives to be open and transparent for both tutor and students. Nevertheless, it is better to keep things in private. For example courses that may involve usage of proprietary software should not be public. To support all kinds of use cases support for private repositories should be added to Flaxo.

Gradle native Junit5 support for generated courses

Consider updating gradle to the latest version and adopt native JUnit5 support.

Or it would be nice to have custom options for JUnit4, JUnit4 as well as gradle version selection on create repo stage.

Course cloning

Sometimes tutor needs to relaunch existing courses. For example, if there are several groups of students then it could be better to separate their solutions into different repositories.

Additionally one may want to launch other person's course.

There are several approaches for courses cloning:

• forking of the repository,
• creating the exact copy of the repository (losing git log history).

Migrate to GitHub GraphQL API

Prerequisites

Currently, GitHub has two api versions:

• v3 - REST API,
• v4 - GraphQL API.

Flaxo uses GitHub API v3 for all operation which leads to a tremendous performance degradation.

Example

On getting pull requests of the repository (~ 70 pull requests) performance differs up to 20 times.

REST API	GraphQL API
85794 ms	4744 ms

Solution

Perform a consequent migration to github v4 API.

Branches adding, deleting, renaming sync

Consider branch sync on renaming.

So, if branch was added, deleted ot renamed, the option should be available to sync it in flaxo.
For the first, let it be available only before course was started, so tutor can customize course before it was started.

Environment generation for docker deploy

Currently, course environment generation feature works only for build from sources deployments. Default docker deployments lacks some local files which environment generation feature requires, f.e. .gradlew file. It ends up with an error on course creation.

Plagiarism analysis fails on files with non-ascii names

It was seen that plagiarism analysis may fail if either tutor or student repositories contain files with non-ascii names.

Course importing via web client

Course importing in Flaxo is available via REST API already. Nevertheless it is not available via Flaxo web client yet.

Graph plagiarism visualization

It is an open question - how to visualize detected plagiarism matches. The view of the plagiarism results can be generated on the server or on the client as well.

Server side solutions

Mossum

https://github.com/hjalti/mossum
Mossum generated graphs images in a variety of formats. Graph shows all matches where percentage is higher than the given threshold.

Client side solutions

d3.js

Dynamic graphs can be generated using d3.js where the percentage threshold can changes according to user preferences.

Сourse closure

Currently, course can be in one of the stages: init and running. It is very natural for course to be closed. F.e. then all deadlines have passed and there are no upcoming pull requests from students.

The tutor should have a possibility to close a course. The course closure means the following:

• course validations should be synchronized by Flaxo,
• active repository listeners such as travis and codacy should be deactivated,
• btw plagiarism analysis and student scores changing should be available.

Suggesting score customization

The suggesting score feature is pretty simple and very helpful at the same time. In order to improve the usage of the suggesting score behavior a customization should be introduced.

A tutor should have a possibility to specify some generic formula for calculating a suggesting score. Auto-completion, syntax highlighting and validation have to be supported.

Do not rely on github profile names

Problem

Currently, github user profile name is used as a user unique identificator which is completely wrong.

F.e. if a student changes his nickname then he will be classified as a new student. Therefore, two students in the task results table can be a single one in the reality.

Solution

Store not only student and user github profile names but also their ids. All identification should based on a github user id rather than profile name.

Plagiarism analysis fails on not found forks

Currently, plagiarism analysis fails in several cases:

• student renames or deletes his/her github account,
• student deletes fork or makes it private.

Flatten user entity to a github account

I can't logout from github for no reason. The button is inactive.

Add webhooks for Codacy validations

Codacy is one of the best static analysis web services. Nevertheless, it doesn't have a strong API. Some time ago several API methods that Flaxo relied on were even removed. Moreover, the service lacks support for webhooks.

At the same time, GitHub provides support for a variety of webhooks including checks. This way Codacy analysis results can be retrieved from GitHub rather than from Codacy itself.

Course repository webhooks can be configured to send information about run checks bringing support for Codacy webhooks to Flaxo!

Plugins system

Flaxo is intended to be flexible and pragmatic in order to maximize an automation while hosting programming courses. Currently, there are several supported solution validations: travis builds, codacy analyses, moss analyses and deadline checks. It is very likely that all validations can be handled generically using some unified interface. Along with solution validations git service providers such as GitHub, GitLab and Bitbucket can be handled in some generic way as well.

The idea behind the issue is creating flaxo platform plugins system. In terms of the system plugin should be a gradle module or a bunch of modules which can be appended to a flaxo distribution at compile time.

Show latest plagiarism analysis timestamp

Currently, there is no marker that would indentify timestamp of the latest plagiarism analysis. However this information can be very helpful to a user.

Moss match percentages evalutation

Problem

Currently, Moss calculates percentage of matching lines in a single plagiarism report based on all lines even if its tutor's pieces of code.

For example, in the given match details it can be seen that all code that was written by the students themselves are classified as a plagiarism. Nevertheless, plagiarism percentage of the given solutions are only 61%. It is most likely that one of the students was using another one's code.

Possible solution

I suppose the natural way to calculate the percentage is to ignore all lines of code that were in the original task.

So the next steps can be:

• Investigate moss submissions run options,
• Configure moss to ignore all tutor's lines of code.

Long running processes

Several user-initiated processes in the system can last from several seconds up to several minutes depending on numbers of students or integrated services involved.

Currently, there is a lack of transparency while working with such processes. Once launched user can retrieve an operation status only after it finishes completely. At the same time, the possibility to see a process progress while it is in progress is quite a valuable feature because it improves user experience significantly. Moreover it increases the system reliability.

The suggesting approach includes the following:

1. Creating a page with a list of long running processes. Each running process entity should include progress bar, status and an option to cancel the process.
2. Creating a common solution for working with asynchronous long running processes with a possibility to observe their status. Probably, some reactive techniques will be of help.

A list of basic long running processes to be supported is the following:

• course creation,
• course starting,
• statistics and reviews refreshing
• plagiarism analysis.

Support for course deletion

There is an API method to delete courses already. It deletes a course entity in the Flaxo database as well as the corresponding repository. The operation is very dangerous because the deleted course cannot be recovered and all its data becomes completely lost.

At the same time the deletion operation can be of help in some specific cases. F.e. if a user has just created a repository with wrong settings it has to be deleted.

In order to support the deletion operation and to minimize risks the delete course button should be added to the course settings. It should have a confirmation popup.

Invalid Codacy projects on course activation

Background

Flaxo uses Codacy public REST API to create Codacy projects and retrieve code static analysis results.

It worked well for some time in the past. But recently Codacy team removed support for an automated projects creation through the API that Flaxo relied on. Notably the corresponding API methods still work but create invalid Codacy projects that cannot be used for automated code static analysis. The only option to create Codacy project today is using an interactive Codacy wizard which is entirely a graphical user interface which is not intended to be used from the server side. Probably, there is a way to overstep such API limitations but it most certainly will be an unreliable approach.

Unfortunately, according to the Codacy support team there is no clear vision on the future of the matter. Therefore, the current implementation will remain until any reasonable solution is found by both Codacy or Flaxo teams.

Manual project recreation

Nevertheless, Flaxo still can be integrated with Codacy. Only one thing has to be done by the tutor to repair the Codacy integration. For every activated course in Flaxo the corresponding Codacy project has to be deleted and then created again with the same name using the Codacy wizard.

Missing webhook for imported course

GitHub webhook is not configured for imported course. Currently, only courses created within Flaxo has configured GitHub webhook.

Mark solutions that were added after plagiarism analysis had been performed

Currently, if user performs plagiarism analysis and new solutions are added after that when there is no way to determine if some solution weren't engaged in plagiarism analysis or no matches were found.

User should definitely see if some solution was not engaged in analysis. Therefore, student record should be highlighted. Also the plagiarism analysis button should be focused somehow. Probably, * can be added to button text: Analyse plagiarism*.

Course statistics in excel format

Support fot xlsx format course statistics export.

Pull request link

It would be nice to have direct link to pull request in task table.

Enable Swagger API documentation

Currently, there is no documentation on the Flaxo's REST API. One of the options to provide REST API documentation is to use Swagger. It provides support for self-documented endpoints which can be of help in the matter.

As the results an additional endpoint api/docs should be created that returns an HTML documentation of the Flaxo's REST API.

Task plagiarism analysis

Problem

Currently, the only option to analyse plagiarism is to launch it for all tasks at once. However it is more common to launch plagiarism analysis for a specific task rather than for a whole course.

Solution

Move analyse plagiarism button to task tab component and update corresponding controller method.

Repair ReadTheDocs build

Custom mkdocs theme material was used in 7786217 which led to a failed documentation build process on readthedocs.io.

Explicit readthedocs configuration should be introduced and a missing custom theme package installation should be specified in the build configuration. More info can be found here.

Flaxo docker images

Two separate docker images should be created for frontend and backend.

Integration tests sources root

Gradle task integrationTest has to be introduced. Junit\spek has to be investigated and configured to use src/integrationTest directory instead of src/test.

Currently part of integration tests are located under src/test which supposed to be unit tests directory. Therefore all integration tests has to be moved to src/integrationTest.

Heap dump analysis

Another possible flaxo external validation is Heap Dump Analyzer.

Any use-cases are welcome.

Solutions approval

It would be good to add a column for a manual check.
Meaning - tutor can review pull request and consider it good, so he can just can hit a checkbox or choose an option something like that just to remember that check was done and its result is good or not.
Or at least it can be a short comment column.

Migration to postgresql

Migration from h2 to postgresql in production is necessary.

GitPlag integration

Prerequisites

Plagiarism analysis in Flaxo is performed using MOSS web service. The possibility to analyse solutions for plagiarism is one of the most valuable platform features. *Huge appreciation to all people behind MOSS, hope it will be open sourced one day.

At the same time, MOSS integration has several disadvantages described below.

First of them is the format of the plagiarism analysis output which is an html page. HTML format is not processing friendly and requires additional actions to perform to retrieve the required data.

Second problem relates not to MOSS web service but its clients. Clients are used to send files for plagiarism analysis to MOSS. Most popular clients take file system level inputs of specific structure. This is a pretty slow mechanism in conjunction with big courses.

GitPlag

GitPlag is a plagiarism analysis self-hosted web service that encapsulates several plagiarism analysis backends and provides both GraphQL and REST APIs.

At the moment the service supports MOSS and JPlag plagiarism analysis backends. This means that integration with the GitPlag brings integration to a whole new plagiarism analysis backend as well!

The platform provides other benefits as well. It is containerized and can be included in the existing deployment infrastructure fluently. Moreover GitPlag uses custom in-memory MOSS client which increases the performance gracefully.

The only problem with the service is the lack of the authentication which can be avoided with the restricted network access.

The following plan is to setup GitPlag as a new self-hosted integrated service. It should replace the MOSS plagiarism validation and becomes part of the Flaxo deployment process.

GitPlag integration checklist:

• Server-side integration.
• Client-side integration.
• Deployment adaptation.
• User documentation on the integration.

Write backend logs to the mounted directory

Currently, backend logs during the docker compose run are written to /opt/logs. Nevertheless, the expected directory for logs is /opt/flaxo/logs which is mounted from the host machine.

The described difference leads to a losing of all backend logs if the corresponding container dies.

Therefore, the configuration has to be altered to store backend logs in the /opt/flaxo/logs directory in the running container.

Add support for custom course generation

Currently, Flaxo can generate java, kotlin and c++ courses and this is quite handy.

On the other hand, current architecture doesn't allow custom courses to be generated. Custom course is the one with unspecified language, testing language or testing framework.

Custom courses creation should be of help if tutor wants to create a course with languages or frameworks that flaxo course generation engine does not support yet.

The following steps should be performed to achieve custom courses creation:

• Course entity language, testing language and testing framework fields should be extracted to an optional course configuration entity.
• Course creation API method should be adapted to the previous changes.
• Client course creation modal should have an additional checkbox. If the checkbox is activated then the language, testing language and testing framework fields should become disabled.

Student role

Currently, the only users of the system are tutors who create and maintain educational programming courses. Initially, there was no need for students to interact with the system. It turned out that student interaction with the system is necessary for several business processes.

First of all, mapping GitHub student profiles logins to their names is out of scope for automation and should be explicitly specified by tutor in some software such as excel or libreoffice. It can be done but it is a huge technical problem because tutor should ask each student for his nickname, write it down, found his nickname in course statistics table and replace it with a real name. Such a routine should be done at the end of each course and it is extremely frustrating. Only if students can fill their personal data all by themselves it will solve the problem at the very beginning.

The second problem lies in the feedback that students receive from the tutor during the course. Students interacts with travis and codacy services where they can see their solution performances. At the same time there is no way for students to see their solution scores which tutor has specified in the flaxo system. The possibility to see solution scores can be of great help for students. Only a plagiarism analysis results should be kept secret to avoid anxiety and chaos between the students.

The third problem comes with an absence of associated metadata for each student. Such as university, group and etc. Such a data can be of help f.e. when courses are hosted for several groups of students.

The solution for all of the problems listed above can be declared in a few steps:

1. Create a student role.
2. Create an authorization for students.
3. Support metadata specifying: first name, last name, university, group and etc.
4. Support personal solution results viewing.

Setup automated development deployment

It will be great to setup an automated deployment to flaxo development server.

Gradle environment generation by gradle init

According to this doc it is pretty easy to generate gradle projects for a bunch of different configurations.

The existing gradle environment generation approach is very complicated and tedious. It will be great to get rid of it and replace it with a predefined set of default gradle init environments.

Code coverage

Coveralls should be configured for Flaxo repository.

By the way travis has some kind of integration with coveralls. It should be inspected as well.

Redesign solution approval mechanism

Summary

Solution approvals was introduced in #4.

It turned out that the original implementation does not suit tutor needs very well.

First of all, a solution can only be in one of two states: approved or disapproved. Therefore, as long as solution has been created it instantly becomes disapproved. It may be confusing and even adverse. F.e. if tutor wants to request changes on a reviewed newest solution he has to mark a solution as approved to be able to mark it as disapproved after.

Moreover, it brings some inconsistency with pull request reviews. Once created a solution has a disapproved status in flaxo but does not have any backing pull request review after it.

The last problem lies in solutions approval UI representation. Only checkboxes are displayed bringing no addition information about the message or the review creation date.

Proposal

Solution approval should have three possible state: none, dissaproved and approved. The last two states can have an associated message and it should be visible for a tutor as well as editable while approval creation.

The idea is to replace a single instanced solution approval with a series of approvals. Each one of the approvals should have a backing pull request review. The series can be shown in the task solutions table after a clicking on a specific solution row. An approval creation button should be right at the end of the approval series.

Wrong course description

There is an option to set course description while creating course. Nevertheless, it doesn't work properly now. Actual course description is replaced with a course name.

Solutions sorting

Add possibility to sort records in course and task tables by one of the columns.

In the result the tables should be sorted by one of the specified column:

• student name by default
• result or score
• build
• code style
• top plagiarism match percent
• deadline
• approving status

User documentation

Currently, there is no user documentation for the system at all. Therefor a first version of the user documentation should be created. The crucial sections to be covered are the following:

• Basic description
• Deployment
• Course creation
• Validations
  • Travis
  • Codacy
  • MOSS

Documentation engine can be mkdocs with some custom theme.

Travis and Codacy report links

Along with #5 Travis and Codacy report links should be available for each solution in the solutions table on a course task tab.

Demo mode of the app

It would be great if the application could run in demo mode. Currently, the only way to test the app is to create multiple GitHub accounts even if a user really doesn't want to use it for real purposes. The possible solution is to show some test data in a fake course and let the user make some interactions with it. Also, there may be some pop-ups to present how the app works.

Add a comment to pull request when it is assessed by a tutor.

As a tutor, I would like to inform student about what score did he get for his solution.
So, to automate that it would be nice if Flaxo add a comment to corresponding pull-request with score when the grade is saved.
It should be async though, that would be pity if it will bloat time required to save grades or even fail saving if commenting failed for any reason.

default .gitignore

Consider adding to all branches default .gitignore file with exceptions with for different IDE-specific files (.idea/**, **.iml, nbproject/**, etc.)

Solution auto-corrections

Summary

Sometimes, students make technical mistakes while working with git and github in specific. It usually is some of the following list:

• Pull request creation for a different branch rather than task branch.
• Extra files indexing in a git, f.e. build/ or .idea.

As long as the mistakes are pretty common for beginners, tutor may spend much time filtering such solutions and requesting changes.

Proposal

The described common mistakes are pretty straight-forward and its detection can be easily automated. Therefore an automated bot response for mistaken solutions can be generated which includes detected problems, ways to solve them and probably even a generated correction.

Automated responses should be an option that can be enabled or disabled in the course settings.

Additional requirement for a described corrector-bot is a customization. All available detecting mistakes should have an option to customize them if there is a possibility. F.e. for git index extra files searching a pattern should be alterable.