ucl-ingi / inginious Goto Github PK

We should be able from the plugin to modify the content of some templates.
For example, we should be able to add a javascript file to the header of all templates.

INGInious could allow students to submit after the deadline.

Downloading all the submissions would not include these late submissions.
Late submissions could be integrated in the valid set (on a per case basis) by the admin.

This allows students to continue to test their code, and the admin to define a custom policy for late submissions.

To be convenient, this topic requires #31 .

Hi,

It would be nice to allow optional inputs.
The workaround I use for now is to ask students to fill them with dummy code lines (e.g. comments) in order to submit their answers.

INGInious does not seem to notify users when a submission exceeds the allowed resources. This is the case when a Java program tries to allocate an array of 4Go in the java7 container (limited to 300Mo). Also, I observe the same behavior with time limits. My task.json file is as follows :

{
    "accessible": true,
    "name": "The Knapsack Problem",
    "limits": {
        "memory": 300,
        "time": 120,
        "output": 1
    },
    "author": "rhartert",
    "problems": {
        "question1": {
            "name": "Upload your solver here",
            "language": "scala, java",
            "header": "Your solver should be contained in a standalone executable jar file.",
            "type": "code-file",
            "allowed_exts": [".jar"],
            "max_size": 31457280
        }
    },
    "environment": "java7",
    "context": "A pretty description of the problem."
}

It would be nice to add tests on the final installation.

These could ensure that

• docker is reachable and up to date
• the tasks folder exists (and is non empty ?)
• some test task can be executed correctly

This would allow users to be reasonably sure that they did not

• forgot to install python packages
• misconfigured some important value
• forgot to install some plugin (a missing json plugin can be very annoying)

More generally, the test suite could be generalized to run locally.
For this to work, we need to be sure that the tests cannot corrupt the current installation.

Some run scripts (probably doing bad stuff) do not stop when the specified time limit is reached. This is the case for instance with the following run script which executes a simple jar file (print the first argument) on the java7 container :

#!/bin/bash

# File to test
$(getinput "question1") >> test.jar

# Runs test script
output=$(java -jar test.jar hello_world)

ID with numbers are not accepted.

All is in the title. This information is important for late submissions. (see #30).

INGInious should allow particular users ("tutors") to see what the students made for different tasks, but should not be allowed to modify tasks or course settings.

Hello folks !
All the run scripts of my tasks have the same library has dependancy. To solve this, I put a copy of the library in each task folder. It would be nice to have a specific folder in which we could put shared resources among tasks. What do you think ?

The following markup :

Title
-----

is interpreted as a level1 header instead of a level2 header. This is not a problem in itself, except that the main task title is rendered as a level2 header (which is confusing).

It would be interesting if INGInious could handle syntax highlighting in the tasks descriptions to help the students in their understanding of the exercise and the piece of program they have to implement by themselves

It would be great and useful if there was a potential success/error message attached to every choice of a multiple choice question. This message would be displayed when the student select this choice in particular. The teacher would be free from adding or not a success/error message to a choice in particular.

Although the success / failed results may be sufficient for many courses, some of them may require a more advanced way to evaluate submissions (e.g. score on 10 points). This problem could be addressed using the archive command but does not provide users (teachers and students) with an elegant way to recover student's grades.

For now, we have always disabled the network in the containers, because it rans untrusted scripts made by teachers AND students.

The new backend (#36) now separates clearly the grading part (that runs scripts made by a teacher or its assistants) from the student part (that runs scripts made by students).

I intended to make the network available by default in every grading container, and to disable it by default in student container; however, it breaks compatibility with old-style tasks (not using run_student). So network is still disabled.

There should be a way to enable network in each container. Steps:

• Add a configuration option in the task.yaml (for the grading container only) (default: false)
• Modify the studio to take care of the new option
• Modify the backend/agent to allow to start the grading container with/without the network, depending on the option
• Modify run_student to accept the new option for the student containers
• Modify the agent to take care of that option too.

Currently it is impossible to download file previously uploaded to resolve a task.

Loading an old submission cause a javascript error.

The grade is wrongly displayed on the right of the page, when a new submission is graded.

When a task reaches the deadline, the students are no more able to see the task.

This raises two concerns

1. Students cannot see their previous submissions, nor check if their last (in-extremis) submission was accepted or not.
2. The progress indicator changes because the task is removed from the course.

We need to distinguish submission deadline and task availability.

The custom subproblem type only allows to create "boxes" in code subproblems.

There should be an interface to do it without editing json code manually.

Hello folks,

To the best of my knowledge, the only way to recover files submitted by a student is to parse the json file returned by INGInious in which submitted files are encoded in base64 text fields. Would it be possible to get the original submitted files with their correct file extensions directly in the archive downloaded from INGInious ?

BSON (Binary JSON) can be used to send non UTF-8 data to the container. It would allow us to avoid to encode/decode binary strings with base64 (see #4).

There is also less spatial overhead in BSON than in JSON, allowing (minor) speed improvements.

When a submission hits the memory limit on the container, it would be nice to have a distinct error message instead of "failed".

I do not know if it is possible to detect this.
In particular, some code may end with a sucessful feedback even though some process was killed for its abusive memory consuption.
In any case, this is not an expected scenario.

This is linked to #28.

It may be useful (but also potentially dangerous) to allow admins to submit as if they were any account. This would allow late submissions, and some flexibility.

This may not be a good idea. You may prefer implementing #30 .

Users should be allowed to kill the grading process of their current submission.
This is useful when they have found an error in their submission and do not want to wait before making a new attempt.

This would also allow to work around bugs where the submission never gets graded and block any further submissions.

With the most recent version of INGInious, I observe the following error when accessing the main page after having submitted some answers to the [LTEST0000] Test tasks : H2G2 > Hello World! task.

Also, the following screenshot shows that the display of the task is not correct. My grade is 0% despite the perfect second submission.

Traceback (most recent call last):
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/application.py", line 239, in process
    return self.handle()
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "/home/gmaudoux/projets/INGInious/frontend/pages/index.py", line 42, in GET
    return self.call_main()
  File "/home/gmaudoux/projets/INGInious/frontend/pages/index.py", line 99, in call_main
    return renderer.main(open_courses, registerable_courses, except_free_last_submissions, registration_status)
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/template.py", line 1020, in template
    return self._base(t(*a, **kw))
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/template.py", line 881, in __call__
    return BaseTemplate.__call__(self, *a, **kw)
  File "/home/gmaudoux/.virtualenvs/inginious/lib/python2.7/site-packages/web/template.py", line 808, in __call__
    return self.t(*a, **kw)
  File "templates/main.html", line 67, in __template__
    $else:
  File "/home/gmaudoux/projets/INGInious/frontend/custom/courses.py", line 133, in get_user_grade
    cache = UserData(username).get_course_data(self.get_id())
  File "/home/gmaudoux/projets/INGInious/frontend/user_data.py", line 69, in get_course_data
    self.username]).get(
  File "/home/gmaudoux/projets/INGInious/frontend/user_data.py", line 111, in get_course_data_for_users
    result["task_grades"] = {dg["taskid"]: dg["grade"] for dg in result["task_grades"] if dg["taskid"] in user_tasks}
  File "/home/gmaudoux/projets/INGInious/frontend/user_data.py", line 111, in <dictcomp>
    result["task_grades"] = {dg["taskid"]: dg["grade"] for dg in result["task_grades"] if dg["taskid"] in user_tasks}
KeyError: 'grade'

127.0.0.1:51774 - - [12/Mar/2015 10:50:45] "HTTP/1.1 GET /" - 500 Internal Server Error

I am downloading all submissions of a task and the file downloaded by the browser is saved with a tgz extension. However the file format is actually a tar.

This issue has already been raised offline. This bug is to track the progress and schedule the task at some milestone.

It seems that upload of binary file fails. Think this a lack of base64 conversion as data is put in the JSON...

It would be really handy to have a way to see what the students see.

The only way to do that (as of now) is to download their submission and read the json encoded feedback.
If the feedback is rich HTML... yes, you guesssed it :-)

Idea for implementation : add a link on each submission to allow seeing it as its owner would.
Adding the Debug Info in that view would be a nice addition.

The api within the sandbox (getinput, parsetemplate) does not provide any way to extract the original filename.

If a file submission box allows both .zip and .tgz, then it is difficult for the "run" script to unarchive the file without the original extension.

It is sometimes useful to resubmit a previous submission.
I see two main cases

1. When a new submission is accepted with a lower score that the previous one.
   In this case, the students will want to resubmit the best submission to ensure that it is always the one downloaded by the "latest sucessful submission" download algorithm.
2. When testing a task, an admin tests his task with the same submission.
   It is annoying to refill the submission form every time. (And it consumes bandwidth on data networks :-)

More generally, any change to the task description should trigger a new run of the existing submissions.

PS: This may not be my best idea ever ;-)

For some courses, students might have to develop and submit advanced applications based on many dependencies (e.g. standalone jar files). However, as observed by GuillaumeDerval, current versions of INGInious seem to restrict the size of uploaded files to 16Mo.

It is nearly impossible to enter reStructuredText in json strings, and there are cases where the user enters a simple string that gets involuntarily evaluated as restructured text.

My point is that in tasj.json, strings should only be plain strings, or html chunks.
In task.rst, everything should be reStructuredText (obviously).

So

1. parsing of strings in task.json should be disabled by default, and
2. Either task.rst or "{context,name,header}IsHTML" should be documented to avoid asking the user to enter reStructuredText values in JSON strings.

What do you think of that ?

For now, INGInious is provided with a simple plugin that is a passive grader for edX XQueue. While this is the simplest implementation possible (< 100 lines of codes), this also comes with some drawbacks:

• Only one input field per exercise
• maximum 30 seconds to give the grading result to edX (limits greatly some courses with heavy tasks)
• Bad syntax highlighting
• Duplicate code for the context of the exercise (that should be indicated in edX and in INGInious if you really use the frontend)
• Need to ask the edX team to create a queue for you, as there is no public interface for this
• Grading weight is difficult to do
• DIfficult to use other plugins
• Not compatible with anything else than edX
• ...

In order to solve this, we need to implement the LTI specification in INGInious (http://www.imsglobal.org/lti/index.html).
If I simplify a bit, implementing LTI would lead (from the student PoV) to have an iframe on edX leading to a page with all the functions of INGInious.

This will also open INGInious to other LMS, like Coursera, or Moodle, with exactly the same code.

It would be nice to have support for custom Dockerfiles. For example, a Dockerfile in the root of the course would define a 'custom' container accessible for all the tasks.

This would avoid asking for an update of the containers when we want to add an executable.

If building container images from Dockerfiles is secure enough, this would be a great usability jump :-)

I'd like to be able to specify which users are allowed to view and submit tasks for my course.
Ideally, I would like to do it by dropping a CSV file into my course git repository. The format of the CSV file is the one I download from the faculty.

This is the file format:
(Skip the first line, e.g. "LINGI2347 en 2014")
Pgrm,Num fac,Nom,Prénom,Noma,Mail,...

The # of columns can vary but the above ones are fixed.

I think from Noma and email address you can recognize a valid UCL / INGI username.

Thanks for implementing this!

More documentation about creating new tasks is needed.

Studio lacks a 'new task' button...:-)

Currently, INGIinious lacks any smart way to limit the number of submissions.

There are many ways to do that. I think of

1. Hard limit (max. X submission(s), maybe exists ?)
2. Soft limit (after X submissions, points start to decrease)
3. Cooling-off period (must wait X minutes after a submission to be allowed to resubmit)
4. Token bucket (coolong-off with some burstiness allowed; corresponds to the idea of max X submissions per period T).

From an educational point of view, 2 is used in coursera, but it is 3 (or its improved version 4) that I like the most.
This would prevent excessive load on the server, and provide a slight bonus to students that start working early on the assignment.

The main goal of this improvement is to avoid students using INGInious as a debugger ;-)

It would be useful to have a superadmin role, rather than indicating explicitly in each course the administators of the platform.

That will also allow to create a "superadmin administration panel" later.

These fields simply don't appear.

That error is caused by a stupid "except" in the default container.

https://github.com/UCL-INGI/INGInious-containers/blob/3c1579070e93c7a169dac18b78363d7d82b36f47/default/bin/INGInious#L121

INGInious should be provided with a simple plugin that allow to register new users and connect to them with the database.

When a task is the result of a collaborative work, inginious should provide a way to indicate that.

The main problem with the current implementation is that all the students of a group can submit individually the same deliverable. This is difficult to detect and leads to extra grading work.
It gets even worse when students of the same group submit different deliverables...

This is very important as many projects/assignements are collaborative.

INGInious should allow to start background tasks that takes as input all the submissions made for the course, and outputs a file or an HTML file.

An use case is for example to do plagiarism detection.