riccap / lodeg

Log analytics for the LODE platform

LoDEg

Log analytics for the LODE platform.

The library: lodegML

lodegML is the name of the library on which the project runs. It exposes a facade API through the class LodegSystem. Please refer to the library documentation.

Requirements

Make sure you have python3.X installed and update pip to the latest version.

$ pip3 install --user --upgrade pip

Next, you can optionally create an isolated environment. This is recommended as it makes it possible to have a different environment for each project (e.g. one for this project), with potentially very different libraries, and different versions:

$ pip3 install --user --upgrade virtualenv
$ virtualenv -p `which python3` env

This creates a new directory called env in the current directory, containing an isolated Python environment based on Python 3. If you installed multiple versions of Python 3 on your system, you can replace `which python3` with the path to the Python executable you prefer to use.

Now you must activate this environment. You will need to run this command every time you want to use this environment.

$ source ./env/bin/activate

Next, use pip to install the required python packages. If you are not using virtualenv, you should add the --user option (alternatively you could install the libraries system-wide, but this will probably require administrator rights, e.g. using sudo pip3 instead of pip3 on Linux).

$ pip3 install --upgrade -r requirements.txt

WebApp

To deploy the WebApp run the following command in the folder lodeg_website:

$ python manage.py runserver [port_number]

The port number is not mandatory, defaults to 8000.

Tests

The following tests are provided:

• Unit tests (with coverage)

  $ make tests [i,interactive]
  

• Library integrity tests (between the WebApp and the standalone library)

  $ make check
  

• Performance tests and benchmarking (todo)

  $ make tests performance (todo)
  

• Logs integrity check

  $ make tests i
  

  and then select System test.

lodeg's Issues

Implement mute intervals extraction

Implement the extraction of info about when the user mutes and unmutes the video.

Display error message when mongo server is down

An error message should be displayed when the library cannot establish a connection with the database.

Automate stress test for the platform

Automate DB migration and script launch to easily run stress tests on the platform.

Acceptance criteria: a script will be provided (or in Makefile) that runs the test population generation script, imports the data into the database and runs the test suite. Each build should provide a report that contains at least:

• memory usage
• clock time
• number of queries launched

New config selections broke LodegSystem methods

Modifying the system configurations does not change method default parameters.

Complete the Contact page in the WebApp

The Contact page in the WebApp does not work as expected. The issue is due to the image upload.

Acceptance criteria: the user gets to the Contact page, fills the form (with the upload of an image) and clicks the submit button. At the end a email has been sent to the web app admin address.

Add methods to the APi for export/impor of the whole system

A way of exporting/importing the system has to be provided. The easiest one is json. Zip can be provided as well.

Run the AWstats webapp

Understand how to make AWstats work on the Apache server.

Add support for large screens in bootstrap columns

When we define divs as bootstrap columns we define only the md (and sm) sizes. Large screens has to be supported too.

Add the capability of saving images to autoplot

The AutoPlot class should provide a way to save figures

Add coverage test to the test suite

Add MakeFile

Add a make file to automate:

• clean
• test
• deployment

Implement registration for students

Implement a registration page for new users.

Add settings page to the WebApp

The library facade interface (the LodegSystem class) is configurable; we will need to provide a way for the administrator to change these configurations in the WebApp.

Migration broke test suite discovery

Unittests does not work after migration due to relative imports; fix imports and ensure that the tests work.

Implement user and password in connection_to_mongo

The authentication in the connection to the database is still missing. We need to provide a way of changing username and password when connection (prompting the user?).

Add a preview of thesis to the documentation

A first preview of the the thesis written on the project will be provided as documentation.

Update charts labels and set default time slot tick to minutes instead of seconds

Add integrity check for real_logs

Before the deployment of the website we need to ensure that the system works nominally; to do so, we need to provide a test suite that checks for records integrity, namely running the whole library and looking for exceptions.

Implement deep key selection in exports

Write the documentation of the library test suite

The library suite test has to be thoroughly documented.

Complete migration from the old git repo

After the migration we will work only on the code of this repository (leaving the old repository.

Move export/import away from LodegSystem class

We should move the implementation of import/export to another file.

Add documentation to import/export in system.py

Implement CSV export from command line

By now LodegSystem.getCSV() works only when used from the WebApp (it returns an HTTP response). We should implement a way of exporting data from command line.

Automate code migration for the extraction library

The extraction library lives in two versions: the stand-alone version and the WebApp one; although they are the same library, the relative imports are different. We need to provide a way of automatically migrating the library from one folder to the other (fixing the imports) and the other way round.

Review how dates and locales are handled in code

Since we use both timestamps and datetimes in code, we need to ensure that the appropriate conversion to locales is worked out on requests.

Add a modal to select filter parameters (course, user, session, lesson)

A modal should be implemented in order to get rid of all the selects that pollute the code. All the interactions should happen via this modal.

Fix the library test suite

Fix the library test suite to use the most recent version of the extraction library. At the moment the test suite has some regression issues introduced when the log format was change.

Fix links in README

It seems that there are some links that still do not work in the project README.

Add documentation of add_interval

Fix regressions introduced into the user section of the WebAbb

The section dedicated to LODE users is suffering some regressions due to some changes made to the lodegML library. For the first release we should provide at least a working preview of this section.

Add the capability of saving images from the WebApp

The WebApp should provide a simple way of saving the images it shows.

Add time_rel/time_abs axis to course user distribution graph

The home user distribution graph is not completed. We need to add the distinction between time_rel and time_abs and test it.

Add plot capabilities without the web interface

The auto_plot library should be configured to plot graphs without the web interface. This can be easily achieved by leaving out the redirection of the function pyplot.plot(); jsut add a configuration param in LodegSystem.

Compute average_session_duration at userInfo level

The average session duration can be another interesting element that can help when clustering user.

Complete the About (Help) section of each page

The help section of each page is still missing content.

Acceptance criteria: the website provides the user with a meaningful and straightforward help section for each main page of the website.

Error when exporting datetimes

TypeError: key datetime.datetime(2017, 9, 6, 16, 26, 41, tzinfo=<bson.tz_util.FixedOffset object at 0x7fbcb7cfcbe0>) is not a string

Workaround = json.dump(skip_keys=True)

Probably the reason is that tzinfo is not an instance of datetime.datetime and the BetterEncoder does not work with it.

Add multithreading to speed up data extraction

Automate lodegML documentation compilation

Add a make doc to the Makefile.

Rename pythonML -> lodegML in documentation

Correct the documentation in order to be compliant with the renaming of the platform library.

Reformat using PEP8

Reformat the code to be compliant with PEP8 specification.

Add impor/export documentation in README

Add the documentation for the impor/export mechanism.

Provide a complete documentation to auto_plot.py

There are some functions that are not documented in auto_plot.py.

Fix missing plot graphs titles and descriptions

Check that all graphs plot at least a title, a description and a legend.

Add integration tests for library code migration

Add an integration test that checks if code migrations of the library has been worked out successfully.

Fix day session distribution view

The day session distribution polar chart ticks are wrong; we can choose between two polar charts displaying 12 hours each, or a polar chart displaying 24 hours (the one we have now) and just correct the ticks labels.

Add missing locale detection to django application

Default locale has been set to CET. The Django framework requires manual detection and tuning of locale time zone in order to convert datetimes to the current locale, operation that is missing at the moment. Add a way to automagically detect locale timezones to the Django App.

Extract_user_data does not work when called directly

Try to understand why extract_user_data does not work when called directly and fix the problem.

Check if caching system works in views

The caching system does not work nominally when admin user is set in django views.

Django keeps data in cache with mem_opt = True

When mem_opt is set to False Django (or MongoDD??) store the last query in cache; this memory is not necessary (at least if you do not want to run the extraction several times in a row); We should clear this cache and restrict data extraction to a maximum number of times to avoid performance issues and memory overflow.