repobee / repobee-junit4

Plugin for RepoBee that runs JUnit4 on cloned repos

Important: I am currently unable to publish any new releases of RepoBee to PyPi. As a result, it's not possible to use repobee manage upgrade to get the latest version. I have opened a request to regain access to my PyPi account over here, but due to the sheer amount of account recovery requests that remain unanswered I have little hope it will be resolved anytime soon.

As a workaround, to upgrade RepoBee to the latest release you may simply run the install script again

Overview

RepoBee is a command line tool that allows teachers and teaching assistants to work with large amounts of student Git repositories on the GitHub, GitLab and Gitea platforms (cloud and self-hosted). The archetypical use case is to automate creation of student repositories based on template repositories, that can contain for example instructions and skeleton code. Given any number of template repositories, creating a copy for each student or group is just one command away. RepoBee also has functionality for updating student repos, batch cloning of student repos, opening, closing and listing issues, no-blind and double-blind peer review, and much more!

In addition, RepoBee features a powerful plugin system that allows users to either use existing plugins, or write their own. Plugins can do a wide range of things, including making RepoBee compatible with multiple hosting platforms (GitHub, GitLab, Gitea), providing compatibility with repositories managed by GitHub Classroom, or running JUnit4 test classes on cloned student repositories.

Still not quite sure what RepoBee actually does? The demo video below briefly explains some of the most important concepts, and showcases how RepoBee can be used to setup and clone student repositories, as well as how to write a simple plugin.

repobee-demo-low-quality.mp4

Short video demonstration of using RepoBee and writing a simple plugin. For a higher-quality version of this demo, click this link!

Feature highlights

RepoBee has a lot going for it. Here are some of the things we are most proud of:

• Compatible with GitHub, GitLab and Gitea: No platform lock-in!
• Support for group assignments (multiple students per repository)
• No-blind and double-blind peer review, directly on the hosting platform
• Generate, clone and update student repositories based on templates
• Open, close and list issues
• Extend and customize RepoBee with the plugin system
• Very little configuration required on the Git service platform side
  • The only requirement is to have an Organization/Group with private repository capabilities!
• No local configuration required
  • Although setting a few defaults is highly recommended

For a full list of RepoBee's built-in (i.e. non-plugin) features, see the user guide and CLI reference.

Getting started

First, either install RepoBee or grab the Docker image. Then, start exploring the Docs, where you (among other things) will find the user guide. It covers use of RepoBee's various commands by way of practical example, and should set you on the right path with little effort.

We also provide a Gitea test instance for playing around around with RepoBee in an environment where messing up has no consequence. See the RepoBee Gitea docs for details on how to use RepoBee with Gitea.

Install

We provide an install script that sets up an isolated and correctly configured environment for RepoBee, granted that you have Python 3.9+ and Git installed. The script supports macOS, Linux and Windows Subsystem for Linux (WSL). You can execute it directly using curl, with either bash or zsh.

IMPORTANT: If you use any other shell than bash or zsh, you must still execute the install script with one of them.

$ bash <(curl -s https://repobee.org/install.sh)

$ zsh <(curl -s https://repobee.org/install.sh)

For additional details, please see the install docs.

Docker

We offer a fully featured Docker image that can be used instead of installing RepoBee.

Versioning

As of December 17th 2018, RepoBee's CLI is a stable release and adheres to Semantic Versioning 2.0.0. The internals of RepoBee do not adhere to this versioning, so using RepoBee as a library is not recommended.

The plugin system is mostly stable as of RepoBee 3.0, but there is a slight risk of breakage due to unforeseen problems. If you develop a plugin, please get in touch so that can be taken into consideration if breaking changes are introduced to the plugin system.

License

This software is licensed under the MIT License. See the LICENSE file for specifics.

Citing RepoBee in an academic context

If you want to reference RepoBee in a paper, please cite the following paper:

Simon Larsén and Richard Glassey. 2019. RepoBee: Developing Tool Support for Courses using Git/GitHub. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '19). ACM, New York, NY, USA, 534-540. DOI: https://doi.org/10.1145/3304221.3319784

repobee-junit4's Issues

Update for repomate-plug 0.2.0

repobee/repobee-plug@ae4c1fc

Refactor the test suite to test the Task instead of the internal methods

It will make it a lot more maintainable.

Classpath should be picked up in constructor

Right now, the classpath is picked up in the config_hook. I'm not sure what the thinking was there, but it has nothing to do with the config file at this time!

Refactor junit4 module

It's starting to become a hot mess, and needs to be refactored into several modules. Off the top of my head, these are the distinct (and relatively unrelated) tasks it currently performs.

1. Finding files.
2. Extracting information from Java files (package name, abstract or not).
3. Compiling Java code.
4. Running unit tests.

That's quite a lot.

Receiving error about classpath

When trying to run
repomate -p junit4 clone -mn WEEK -rtd TEST-DIRECTORY -v
I receive
[ERROR] _ActException: HookResult(hook='junit4', status=<Status.ERROR: 'error'>, msg='junit-4.12.jar not on the classpath')
Even though the classpath is set correctly

Replace HookResult with Result

repobee_plug.HookResult is deprecated.

Add a user-configurable timeout

Even if the test class does not include timeouts, the plugin should timeout after a while. This can default to 10 seconds or so, that should be sufficient for the vast majority of test cases.

The problem with this method is that it can't be set on the granularity of single tests, this will be a timeout for running all test classes.

It should be possible to set the timeout both on the command line and in the config file.

Remove Python 3.5 support

Same as repobee/repobee-feedback#7

Redo how test results are displayed

Should show the following:

• Amount of tests passed, and amount of total tests for each test class
• Amount of tests passed, and amount of total tests for all test classes combined
• Compile errors separately

Furthermore, the following statuses should be used:

• All is good: plug.Status.SUCCESS
• Test failure: plug.Status.WARNING
• Tests couldn't run (e.g. compile error): plug.Status.ERROR

Place dependency on current version of repobee

repobee-junit4 should depend on the core repobee package, such that one can't use repobee-junit4 with a version of repobee that is too old. Each release should set this dependency to the latest stable version of repobee.

Should run java with the `-ea` flag to enable plain asserts

There's no good reason to run tests without plain asserts enabled.

Update docs with some example uses

The documentation is a bit vague on the use case of the plugin

Should check that supplied rtd exists

Add support for packages

If test classes with package statements are found in the rtd, should try to run source code as packages.

Since the package statement must be the first line in the file, it's near-trivial to find out if a source file is packaged or not.

Factor out all Java-related stuff to a separate module

See #16

Run the Mac builds with TOX instead of multiple travis instances.

much of the time is taken up by just updating homebrew, so this will probably save a lot of time on builds.

Port to RepoBee 3

Adapt to the new task-based plugin system

Add links/explanation of hamcrest and junit4 jars

They just kind of appear in the configuration, should be mentioned in install.

Refactor the destination of CLI options

Currently, the junit4 prefix is removed in the destination, which could still cause clashes with other plugins. All of the destinations should be prefixed with junit4_, and then there could simply be an intermediate step of translation in the parse_args method that strips the prefixes.

Add continuous deployment

Update dependencies

Build fails because of version mismatch

Should require pytest>=4.0.0 and pluggy>=0.8.0

Rename to repobee-junit4

Move documentation to RTD

The docs are getting large enough that it's about time to move to RTD. #3 will make the docs even larger.

Use branch coverage in CI

The Travis run script is set to statement coverage

Test on Python 3.8

Truncate long lines for `-v` and add `-vv`/`--very-verbose`

-v currently gives completely untruncated output. This is a bit inconvenient when you just wanna see the tests that fail, and not necessarily tons and tons of output with that. Two changes are necessary:

1. Rename -v|--verbose to -vv|--very-verbose
2. Add a new -v that truncates long lines.

Add Flake8 to CI

Early exit on bad Flake8

Plugin should not raise mid-execution

This plugin sometimes raises an _ActException if something goes wrong in a repo. This should not happen, spontaneous errors should be reported to RepoBee through hook results. Only errors that occur immediately (such as there not being a reference tests directory) should actually cause an error to be raised to RepoBee itself.

Update documentation with new CLI options

Should reflect the changes made in #54

May also be a good idea to put up a notice in the README.

Adding a way to run student-written tests while cloning

It would be nice to able to run the student-generated tests in every repo using repobee

ExceptionInfo.__str__ from pytest.raises no longer yields exception message

See repobee/repobee#257

Allow Junit4 to inspect the stack trace

The policy should allow junit4 the RuntimePermission.getStackTrace permission, or it won't be able to show stack traces for crashes.

Test suite out of date with pytest/pluggy

Sigh... pytest_namespace has been removed from pytest as of 4.0.0

Truncation should also affect amount of lines

Compile errors generate a ton of lines, that should only be shown with the --junit4-very-verbose option. Otherwise, truncate to ~5 lines.

Add pre-commit hooks

For running black, trimming whitespace and inspecting YAML to begin with.

Comply with flake8

Before doing #36 and #19

Add verbose option

Add a -v|--verboseoption to output more info on test failures.

Should ignore abstract test classes

Currently tries to run abstract test classes, which does not work out very well

Prefix options with `--junit4`

See repobee/repobee#388

Add possibility to specify security policy

The JVM has a built-in security policy manager. A policy can be specified from the command line. It would probably be good to have a default policy which is pretty restrictive, and have the possibility to specify a custom policy.

Bad regex in `setup.py`

See repobee/repobee-plug#7

Clean up Java `.class` files between every integration test

Most integration tests compile .java files to .class files, and these are not cleaned up between runs, which could potentially lead to some trouble.

Docs still reference Repomate

Update to RepoBee!

Refactor test suite

It's getting a bit messy.

Add default, strict security policy

Related to #22, a first step that adds a default, strict security policy that just disallows... everything.

Remove hamcrest jar as a strict dependency

Strangely enough, not everyone uses the hamcrest matchers.

Update interactions with HookResult

repobee/repobee-plug#26 will break at least one iteration over hook results.

Run mac build only for 3.5.4

It runs way too slowly, testing for 3.5.4 should be sufficient

Poor user feedback if any of the jar files (junit or hamcrest) do not exist

It just results in compile errors as symbols from those jars can't be resolved. Should probably raise an exception if the jars can't be found at the latest time they need to be found at the locations the user specifies.