jupyter / nbgrader Goto Github PK

A system for assigning and grading notebooks

Home Page: https://nbgrader.readthedocs.io/

License: BSD 3-Clause "New" or "Revised" License

Python 37.21% JavaScript 6.76% CSS 0.56% Smarty 0.63% Jupyter Notebook 31.39% Batchfile 0.22% Makefile 0.21% HTML 16.55% Mako 0.01% TeX 0.13% Jinja 0.50% TypeScript 5.82% Shell 0.02%

nbgrader jupyter jupyter-notebook jupyterhub teaching grading

nbgrader's Introduction

nbgrader

Build: Forum: Cite:

A system for assigning and grading Jupyter notebooks.

Documentation can be found on Read the Docs.

Highlights of nbgrader

Instructor toolbar extension for Jupyter notebooks

The nbgrader toolbar extension for Jupyter notebooks guides the instructor through assignment and grading tasks using the familiar Jupyter notebook interface.

Instructor "formgrader" extension for Jupyter notebooks

The formgrader extension for the Jupyter notebook allows instructors to use the core functionality of nbgrader---generating the student version of an assignment, releasing assignments to students, collecting assignments, autograding submissions, and manually grading submissions.

Student assignment list extension for Jupyter notebooks

Using the assignment list extension, students may conveniently view, fetch, submit, and validate their assignments.

The command line tools of nbgrader

Command line tools offer an efficient way for the instructor to generate, assign, release, collect, and grade notebooks.

SciPy 2017 overview talk (click to view on YouTube)

Installation

For detailed instructions on installing nbgrader and the nbgrader extensions for Jupyter notebook, please see Installation section in the User Guide.

Contributing

Please see the contributing guidelines and documentation.

If you want to develop features for nbgrader, please follow the development installation instructions.

nbgrader's People

Contributors

Stargazers

Watchers

Forkers

ellisonbg jhamrick bwlv tasilb scshepard carreau jdfreder redslug willingc haraldschilly alope107 lphk92 ssanderson ellachao svurens modulexcite dementrock minrk vinaykola gpfreitas danielballan lazymike davis68 parente bioh4x silky acourtney2015 michigraber quevedin drfrankied norivicjr c0ns0le booox lgpage angelapper thehrh scientific-coder cn-upb chakan-num dhanus gm1225 edwardjkim xiaosongz jharbig adamchainz yonghuming choldgraf firdouskhan stuartf whitead ttamayo charnpreetsingh abhishekbajpayee alexxnica oarriaga enterstudio elhele rameshjes deebuls alexewd lfmartins jilljenn jonduan jhillgithub alchemyst madhu94 enriquebrgn ixjlyons darthdippy ruin0x11 ksunden kmfolgar ningwenfei mpacer alexanderallenbrown tedeler thotypous kiranvasudev randy3k vidartf jennhamrick zonca sashabaranov rwest vahtras johnhw apachipa consideratio guoyu07 twfenwick makhalin fabianoelscher ghego jalpan-randeri gennad-gentoo www3838438 sigurdurb terhorst ryanlovett fperez

nbgrader's Issues

No tests?

It's passing, but I can't actually use it.

Can't create a multiline response

When I hit enter to start a new line, my whole response disappears. I'm able to paste in a multiline response just fine.

Improved navigation within grading a notebook

Right now there isn't a quick way to jump around the notebook. It seems like it would be really helpful to have a floating hierarchical representation that stays put as you scroll down, with hyperlinks to headers, subheaders, and graded cells.

This could be useful in the notebooks more generally (e.g. when students are doing them) but I think that they proceed through them largely linearly, so the real benefit is for graders who need to jump around a lot in the notebooks.

Add documentation

Currently the only docs are the ones that are in the readme. I can set up some docs with sphinx, and then we could put them on readthedocs.org.

Add an option to make grader blind to which student is being graded

This can help eliminate well-known gender and race biases.

Reorganize the various nbgrader apps

They're currently a bit disorganized, and there's some code duplication that I'm unhappy about (e.g. overriding the ipython_dir trait). This is not a major issue, but should be addressed at some point.

Add student view

Currently, you can only see an assignment view (e.g. assignments -> notebooks -> students -> grade). It would be nice to additionally have a student view, so you could do students -> assignments -> notebooks -> grade. This would make it easy to see what a student's grade is for the entire assignment, and what their overall grade is for the course.

Yellow cell background doesn't go away when celltoolbar is disabled

If the "Create Assignment" toolbar is selected, and then deselected, the yellow background for grade cells does not go away.

Round floats for average score

Long floats in the average scores reported for a problem set, e.g. 4.340686274509804 out of 4.5. This should be to just 2-3 decimal places.

Add option to randomize order of students when grading

This will help the Zwirlings of the world, who suffer when graded by the weary eyes of people who graded the Aarons and the Aardvarks first.

nbgrader submit should save timestamps with UTC

Currently it just uses the timezone of the server.

Manual grading in live notebooks

After doing lots of manual grading of notebooks, I have found that doing so in the static HTML nbgrader web app doesn't work. A few issues I ran into:

I found that I often wanted a new comment space in a location where the nbgrader web app didn't put one.
Many comment areas in the nbgrader web app went unused, making it difficult to navigate with all of that extra space.
I also found myself adding code comments in student code cells so I could be very specific like # <---- this line has an index error.
I ended up rerunning student code most of the time. It was very unusual to manually grade a notebook were I didn't rerun something.

Because of this I ended up doing all of my manual grading in the live notebook with a modified nbgrader JS extension that allowed me to simple set the manual grade score (also showed the autograde score). This worked very well.

Tab between grade_id and points in extension toolbar

If you are editing the id and points for a grade cell, and have the cursor in the grade_id textbox, and then press tab, the cell body is selected rather than the points textbox. My intuition is that it should go to the points textbox next.

Use comment syntax instead of jinja templates to denote solution code

In the master version, instructors may want to mark some code inline as solution-only, like this:

def some_function():
    ### BEGIN SOLUTION
    print("this is the solution!")
    ### END SOLUTION

Previously I did this with jinja, but @ellisonbg and I agreed that using comments is better because then the cells are pure python and can still be run without using nbgrader, and it's a clearer syntax for people who are unfamiliar with jinja.

Final value not updated after navigating to next student

If you specify some score in the text box, then click the link to go onto the last student, that last value entered is not preserved. No issues if you click somewhere else on the page, then click the navigation link.

nbgrader doesn't catch if a student turns a grade code cell into a markdown cell

New nbgrader record subcommand

I have found that it caused problems to have the autograde step also record the grades in the db. I think there should be an nbgrader record subcommand that only records the grades that are in the notebook in the db. A side effect of this is that the autograde command should only do the autograding and save the autograde score in the notebook metadata itself. This creates a more modular and loosly coupled system.

Empty cell toolbar

After installing nbgrader I get a notebook with an empty cell toolbar.
The error message is (I am using IPython v 2.3):

'Full credit' and 'No credit' buttons on grading bar

In addition to being able to add the numerical score, there should be two buttons that assign full credit and no credit. Default tabbing to the numerical score still makes sense though.

Prompt numbers display as 'null'

After running nbgrader assign, the prompt numbers in the resulting notebook are null. This may be a bug in the ClearOutputPreprocessor in IPython, but I need to investigate further.

Make grading cells undeletable

As soon as ipython/ipython#6488 is merged, I'd like to provide some way of making the grading cells undeletable. This could either be something that's automatically done with the nbextension, or it could be a configurable flag in the ClearSolutions preprocessor.

If grading cells are undeletable, then we can be (relatively) confident that the cells will still be there when the student turns in the notebook. We can then do things like assign a hash of the cell to make sure the tests haven't changed, or just replace the cell contents with the original test, etc.

Uncaught TypeError: undefined is not a function

Occurs occasionally when loading the extension @jhamrick

Add option to `nbgrader assign` to make a tarball or zipfile

Probably many people will want to tar or zip up their assignments in order to release them. It would be nice if nbgrader assign could (optionally) do this automatically.

Add "mark graded" button

It should be possible to mark a submission as fully graded, so that if a human grader has to stop grading in the middle, they can come back and easily know where they left off.

Autofill non-reponse feedback.

It might be nice to automatically fill in some standard text as feedback for when a student's response is blank, e.g., ("No credit assigned because you did not provide a response.")

Prompt numbers in student version are `None`

For example: http://nbviewer.ipython.org/github/jupyter/nbgrader/blob/master/docs/user_guide/release_example/StudentNotebook.ipynb

This is a problem with IPython rather than nbgrader (see ipython/ipython#6702) but I'm creating an issue here too for reference.

Add a warning/error for code cells that are marked as both grade cells and solution cells

The only cells that are marked as both grade and solution should be cells that need to be graded manually, e.g. markdown cells. I can see some use cases for doing this with code cells as well, so maybe there shouldn't be an actual error, but at least a warning in nbgrader assign because probably in both cases if a code cell is both a grade cell and a solution cell, it's a mistake.

Incompatible with nbformat 4 changes

Currently, nbgrader appears to be incompatible with the new nbformat 4 changes. For the time being, I've clamped the IPython version to be before nbformat 4 was merged (see #28), but we should actually fix nbgrader so that it's compatible with these new changes.

KeyError in autograde and formgrade apps

Trying to run the command line help for the autograde or formgrade apps results in a long traceback:

$ nbgrader autograde --help
Autograde a notebook by running it

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--stdout
    Write notebook output to stdout instead of files.
--init
    Initialize profile with default config files.  This is equivalent
    to running `ipython profile create <profile>` prior to startup.
--quiet
    set log level to logging.CRITICAL (minimize logging output)
--debug
    set log level to logging.DEBUG (maximize logging output)
Traceback (most recent call last):
  File "/usr/local/bin/nbgrader", line 12, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/Users/jhamrick/project/tools/nbgrader/scripts/nbgrader", line 4, in <module>
    main()
  File "/Users/jhamrick/project/tools/nbgrader/nbgrader/apps/nbgraderapp.py", line 61, in main
    NBGraderApp.launch_instance()
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 548, in launch_instance
    app.initialize(argv)
  File "<string>", line 2, in initialize
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/nbgrader/nbgrader/apps/nbgraderapp.py", line 53, in initialize
    super(NBGraderApp,self).initialize(argv)
  File "<string>", line 2, in initialize
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/ipython/IPython/core/application.py", line 378, in initialize
    self.parse_command_line(argv)
  File "<string>", line 2, in parse_command_line
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 460, in parse_command_line
    return self.initialize_subcommand(subc, subargv)
  File "<string>", line 2, in initialize_subcommand
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 398, in initialize_subcommand
    self.subapp.initialize(argv)
  File "<string>", line 2, in initialize
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/nbgrader/nbgrader/apps/customnbconvertapp.py", line 41, in initialize
    super(CustomNbConvertApp,self).initialize(argv)
  File "<string>", line 2, in initialize
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/ipython/IPython/nbconvert/nbconvertapp.py", line 191, in initialize
    super(NbConvertApp, self).initialize(argv)
  File "<string>", line 2, in initialize
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/ipython/IPython/core/application.py", line 378, in initialize
    self.parse_command_line(argv)
  File "<string>", line 2, in parse_command_line
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 472, in parse_command_line
    self.print_help('--help-all' in interpreted_argv)
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 331, in print_help
    self.print_options()
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 302, in print_options
    self.print_alias_help()
  File "/Users/jhamrick/project/tools/ipython/IPython/config/application.py", line 266, in print_alias_help
    cls = classdict[classname]
KeyError: 'FindStudentID'

It seems like it doesn't know about the FindStudentID class, but I don't know why.

Allow adding assignments through the web interface

New assignments could be created through the web interface -- this would mostly just include adding their id and duedate.

Tabbing should bring you to the top of the answer key, not the student's answer

For Markdown cells, allow double click or hover to show non-rendered source

You can get the source through the context menu, but it would be nice to have a way to quickly toggle between the rendering and the source – it helps diagnose what errors were made.

Make student's answer a live, editable cell

When a student get something wrong, I'd like to be able to run their code to help me diagnose what went wrong.

Images are not properly handled by the formgrader

Complexity costs

The nbgrader system is complex enough that when something goes wrong, the average instructor won't have any idea how to recover without writing lots of custom python code. I found myself doing this time and time again, and that isn't sustainable over the long run. I think we really need to figure out how to make it simpler and also more bullet proof.

Use tornado instead of flask

The rest of IPython/Jupyter stuff uses tornado, so it might be better to switch to that from Flask in order to be a bit more consistent. Though, if we end up switching to live notebook grading, we wouldn't need to worry about this anyway. I'm creating this as an issue anyway just as a reminder in case the HTML grader does stick around.

nbgranger assign should abort if there is a graded problem with no points

An issue that I found a couple of times is that I forgot to add points to a problem before I send it out to students. The problem is a complete pain to fix once you get student work back. If a cell is marked for grading, nbgrader assign should abort with an error message saying that the points is empty.

Use SQLAlchemy for DB layer

Originally, it was not clear what the database would end up looking like, so mongodb made a lot of sense. However, seeing how the db tables/collections have evolved, I think it will really help to use sqlalchemy's ORM layer. This would remove the need for us to write and maintain our own custom ORM layer for MongoDB and would allow a much larger set of DB engines, most importantly file based ones that are easier to manage such as sqlite.

Match horizontal position of answer key and student's answers

Alignment cuts down on eye movements and speeds grading.

Minor bugs in nbgrader nbextension

Am using this for the first time (yeah!) and found the following bugs:

The points field is only writen on keyup, but should also trigger on change as there are other ways of changing this field.
Both grader_id and grade_id are being written to the metadata.

Add support for late assignments

Currently, there is no way to track when assignments are turned in, whether they are late, and if so, what late penalty should be applied.

Autograding markdown cells

This is very much a wishlist item, but it would be awesome to somehow be able to autograde markdown cells that have simple and clear solutions.

Make tabbing after final comment cell focus on "Next →"

(I hope these aren't nitpicking. This is a pretty awesome system and the tweaks would make it even better.)

Create a "submit" extension or magic

I'm not entirely sure what the form of this will be, but perhaps a magic where they run:

%submit

and it figures out their username, and the time stamp, and creates a tar archive (or something) with the relevant information. Perhaps it could even then send that tar ball to a data store somewhere.

I think this might be easy to hack -- i.e., easy for students to change the timestamps -- but at least, for a first pass, I think something like this should be sufficient.

Save result of autograding somewhere

Currently, the autograder doesn't do much beyond just executing the cells. It should actually determine which grade cells pass and which don't, and record that information (along with the number of points earned) somewhere -- either CSV, JSON, or a database, perhaps.

Autograder should time out and continue running

It will eventually time out currently, but it will throw an error and halt. It should be smart enough to time out on a particular cell, but continue running on subsequent cells even if it times out.

Compute checksums of grade cells

We should create checksums of the contents of grade cells, and then verify them during autograding to ensure the contents haven't changed. Probably all this will require is a config option to specify a salt value, and then the instructor will need to ensure that the salt is the same for both nbgrader assign and nbgrader autograde.

Display total score in create assignment toolbar

I typically want my assignments to have some number of points across all the problems -- e.g., 20 points -- but it's hard to count up the points for all the individual parts. It would be nice if at least each notebook said how many points it was worth at the top or something.

Raise an error if the same grade_id is used in the nbextension

autograde not working

I get the following output:
Any help is appreciated.

Ivan

!nbgrader autograde

[AutogradeApp] Using existing profile dir: '/tmp/nbgrader/profile_default'
Traceback (most recent call last):
  File "/home/ivan/.virtualenvs/nbgrader/bin/nbgrader", line 4, in <module>
    main()
  File "/home/ivan/.virtualenvs/nbgrader/lib/python3.4/site-packages/nbgrader/apps/nbgraderapp.py", line 71, in main
    NBGraderApp.launch_instance()
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 563, in launch_instance
    app.initialize(argv)
  File "<string>", line 2, in initialize
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/ivan/.virtualenvs/nbgrader/lib/python3.4/site-packages/nbgrader/apps/nbgraderapp.py", line 63, in initialize
    super(NBGraderApp, self).initialize(argv)
  File "<string>", line 2, in initialize
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/core/application.py", line 368, in initialize
    self.parse_command_line(argv)
  File "<string>", line 2, in parse_command_line
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 470, in parse_command_line
    return self.initialize_subcommand(subc, subargv)
  File "<string>", line 2, in initialize_subcommand
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 408, in initialize_subcommand
    self.subapp.initialize(argv)
  File "<string>", line 2, in initialize
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/ivan/.virtualenvs/nbgrader/lib/python3.4/site-packages/nbgrader/apps/customnbconvertapp.py", line 54, in initialize
    super(CustomNbConvertApp, self).initialize(argv)
  File "<string>", line 2, in initialize
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/config/application.py", line 74, in catch_config_error
    return method(app, *args, **kwargs)
  File "/home/ivan/.virtualenvs/nbgrader/ipython/IPython/nbconvert/nbconvertapp.py", line 197, in initialize
    self.init_notebooks()
  File "/home/ivan/.virtualenvs/nbgrader/lib/python3.4/site-packages/nbgrader/apps/customnbconvertapp.py", line 98, in init_notebooks
    self.prefix = os.path.commonprefix(zip(*self.notebooks)[0])
TypeError: 'zip' object is not subscriptable

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at [email protected]

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    c.Application.verbose_crash=True