GatorGrader

The only tool you'll need to ensure your student's code and writing is up to speed!

Quickstart Guide
Key Features
What Do People Think about GatorGrader?
Installing GatorGrader
Testing GatorGrader
Running GatorGrader
Comparison to Other Tools
Presentations
Contributing
Contributors

Quickstart Guide

Want to quickly get up and running with GatorGrader? If you have an existing assignment and you want to automatically check its source code or technical writing, then you can use GatorGrader's Gradle plugin. The following starter repositories provide examples of how to configure GatorGrader to check programs and documentation for different languages: Java, LaTeX, and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window.

Key Features

GatorGrader automatically checks the work of technical writers and programmers. It can:

Use its Gradle plugin to check projects implemented and documented in a wide variety of languages (e.g., Java, Python, LaTeX, Markdown, HTML, and CSS).
Integrate with GitHub Classroom to check solution and starter repositories created for professors and students, respectively.
Run in a cloud-based environment like Travis CI or on the command-line of a developer's workstation.
Operate as a "batteries included" grading tool, supporting automated checks like the following:
- Does a file exist in the correct directory with the requested name?
- Does technical writing contain the desired number of words and paragraphs?
- Does source code contain the designated number of language-specific comments?
- Does source code or technical writing contain a required fragment or match a specified regular expression?
- Does a command execute correctly and produce the expected number of output lines?
- Does a command execute and produce output containing a fragment or matching a regular expression?
- Does a GitHub repository contain the anticipated number of commits?

Aligning with key recommendations in a recent National Academies report, GatorGrader helps instructors automatically check student submissions in both introductory and application-oriented classes using languages like Markdown, Java, Python, JavaScript, CSS, and HTML. GatorGrader does not aim to solve problems related to building and linting a project or managing an assignment's submission, instead integrating with existing tools and systems like Gradle, GitHub, and GitHub Classroom to effectively handle those tasks.

Installing GatorGrader

Installing GatorGrader is not necessary if you intend to use it through its Gradle plugin. If you want to participate in the development of GatorGrader, the project maintainers suggest the use of Pyenv to install Python 3.6 or above. In addition to installing Git to access the project's GitHub repository, you should also install Pipenv for its support of package and virtual environment management. After completing the installation of these tools, you can type the following command in your terminal window to clone GatorGrader's GitHub repository:

git clone https://github.com/GatorEducator/gatorgrader.git

If you plan to develop new features for GatorGrader or if you want to run the tool's test suite in Pytest, then you will need to install the developer dependencies by typing pipenv install --python="$(pyenv which python)" --dev in the directory that contains GatorGrader. If you want to use GatorGrader, then you can type pipenv install --python="$(pyenv which python)" instead. Once these commands complete successfully, you have officially installed GatorGrader! Note that running these commands will ensure that Pipenv creates a virtual environment for GatorGrader that is bound to the version of Python that Pyenv set for use.

Testing GatorGrader

Automated Testing

The developers use Pytest for the testing of GatorGrader. Depending on your goals, there are several different configurations in which you can run the provided test suite. If you want to run the test suite to see if the test cases are passing, then running this command in a terminal window will perform testing with the version of Python to which Pipenv's virtual environment is currently bound.

pipenv run test

Test Coverage

Along with running the test suite, the developers of GatorGrader use statement and branch coverage to enhance their testing activities. To see the coverage of the tests while also highlighting the lines that are not currently covered by the tests, you can run this command in a terminal window. As with the previous command, this will run the tests in the version of Python to which Pipenv's virtual environment is currently bound.

pipenv run cover

Testing with Multiple Python Versions

The previous two commands are restricted to running the test suite in the version of Python to which Pipenv was bound. If you have installed multiple versions of Python with Pyenv and you want to iteratively bind Pipenv to each version and then run the test suite, then you should first run the following commands to install Pipx and use Pipx to install Invoke. The first of these three commands will install pipx, a program that supports the execution of Python packages in isolated environments. The second command makes the directory ~/.local/bin/ a part of the search path for executable Python programs and the third one installs the invoke command so that it is available on your workstation outside of a virtual environment managed by Pipenv, thereby ensuring that it is always available to run tasks.

pip install pipx --user
python -m userpath append ~/.local/bin/
pipx install invoke

Now you can run the test suite in the specified versions of Python with the following command. This example command will run the test suite in Python 3.6.8 and Python 3.7.3.

invoke -c scripts/tasks test --pyenv 3.6.8 --pyenv 3.7.3

If you want to track test coverage while running the tests in both Python 3.6.8 and 3.7.3, then you can run the following command.

invoke -c scripts/tasks cover --pyenv 3.6.8 --pyenv 3.7.3

You can switch the version to which Pipenv is bound by running the following command that adopts, for instance, Python 3.7.3.

invoke -c scripts/tasks switch --pyenv 3.7.3

Code Linting

The developers of GatorGrader use linting and code formatting tools, such as Pylint, Pydocstyle, and Black. After installing GatorGrader's development dependencies with Pipenv, you can run all of the linters by typing this command in a terminal window.

pipenv run lint --check

Linting Checks

Want to learn about our linting checks? Check us out on our website, GatorGrader! We have detailed descriptions of our linting checks and more! To get an idea of the linting checks we offer, here is a quick list:

ConfirmFileExists
CountCommandOutput
CountCommits
CountFileLines
CountFileParagraphs
CountFileWords
CountMarkdownTags
CountMultipleLineComments
CountParagraphWords
CountSingleLineComments
ExecuteCommand
MatchCommandFragment
MatchCommandRegex
MatchFileFragment
MatchFileRegex

Something you should know when working with our linting checks is that all of them come with optional arguments. These optional arguments are: -h, --help. The help message will be displayed and then exited. If further assistance is needed, please contact us on GitHub.

Running GatorGrader

Students and instructors normally use GatorGrader through its Gradle plugin, specifying the requested checks in a config/gatorgrader.yml file. When run through Gradle, GatorGrader reports each check that it performed, additionally sharing a diagnostic message for each check that did not pass. Individuals who want to run GatorGrader as a stand-alone Python application should first install it's application dependencies with Pipenv and then learn about the supported checks and their defaults by typing pipenv run python3 gatorgrader.py --help in a terminal window.

Instructors often run GatorGrader in conjunction with other tools that check source code and technical writing. For instance, in a Java-based introductory course, instructors could verify student submissions with Checkstyle, thereby ensuring that the Java source code adheres to the requirements in the Google Java Style Guide. In this course, an instructor could require that Markdown files with technical writing meet the standards described in the Markdown Syntax Guide, meaning that all Markdown files must pass the checks performed by the Markdown linting tool. These assignments could also require that all submitted technical writing must adhere to the standards set by the Proselint tool. Since GatorGrader can run an arbitrary command and check its error code, it is also possible to integrate it with a wide variety of other linters, code formatters, and testing tools.

Comparison to Other Tools

Other automated grading tools include:

autograde-github-classroom: "scripts to download and grade submissions to Github Classroom"
check50: "a tool for checking student code"
Classroom Assistant: "desktop application to help you get student repositories for grading"
nbgrader: "a system for assigning and grading notebooks"
nerfherder: "scripts for automating grading with GitHub Classroom and Moodle"
Submitty: "homework submission, automated grading, and TA grading system"
WebCat: "all-in-one plugin for full processing and feedback generation"

Designed for instructors who want an alternative to simple scripts or stand-alone platforms that do not integrate with industry-standard infrastructure like GitHub and Travis CI, GatorGrader is a tool that automatically checks the work of technical writers and programmers. Unlike other systems, GatorGrader provides a "batteries included" approach to automated grading that makes it easy for instructors to combine well-tested checks for projects that students implement in a wide variety of programming languages. GatorGrader's developers take its engineering seriously, maintaining standards-compliant source code, a test suite with 100% statement and branch coverage, and top-notch source code and user documentation.

Presentations

GatorGrader's creators give presentations about the development, use, and assessment of the tool. Please contact one of the developers if you would like to feature a presentation about GatorGrader at your technical conference. The following list includes some of our team's recent presentations:

A Hands-on Guide to Teaching Programming with GitHub, Travis CI, and Python
at PyOhio 2018
Using GitHub, Travis CI, and Python to Introduce Collaborative Software Development
at PyCon Education Summit 2018
Using Python, Travis CI, and GitHub to Effectively Teach Programming
at PyGotham 2018

Contributing

Are you interested in contributing to GatorGrader, GatorGradle, or any of the sample assignments (e.g., Java, LaTeX, or HTML with CSS)? Great, because we appreciate the involvement of new contributors! Before you raise an issue or start to make a contribution to GatorGrader's repository, we ask that you review the project's code of conduct and the contribution guidelines.

Contributors

Thanks goes to these wonderful people (emoji key):

_{Gregory M. Kapfhammer} 📢 💻 📖 🎨 🚇 ⚠️	_{Saejin Mahlau-Heinert} 💻 📖 🚇 👀	_{Christian Walker} 🖋	_{Andrew Everitt} 💻 📖	_{Christian Lussier} 💻 📖	_{Simon Burrows} 💻	_{Austin Bristol} 💻	_Matt 💻 📖	_{Christopher Miller} 💻 🎨 📖	_{Spencer Huang} 💻 📖
_{Mohammad Khan} 💻	_{Zachary Shaffer} 💻 📖	_{Alexander Yarkosky} 🐛 📖	_Dillon 📖	_{Zachary Leonardo} 💻	_{Jonathan W. Mendez} 💻 📖	_{Tyler Lyle} 💻	_finneyj2 💻 📖	_schultzh 🐛 💻 📖 👀	_alexheinle 🐛 💻 📖 👀
_{Zachary Andrews} 💻 🎨 📖	_{Nicholas Tocci} 📖	_{Devin Ho} 📖	_{Matthew Baldeosingh} 📖	_{Jordan Durci} 💻	_{Karol Vargas} 📖	_{Jerfenson Cerda Mejia} 💻 📖	_{Tara Douglass} 💻 📖	_{Alexander Butler} 💻	_corlettim 🐛 💻 📖 👀
_{Carson Quigley} 💻	_{Joshua Yee} 💻 📖	_{Madelyn Kapfhammer} 📖 🚇 ⚠️