bluepichu / batch-tester

A batch tester for programming problems.

Batch Tester

If you're like me, you do a lot of programming competitions. If you don't want a bunch of incorrect submissions, these require some method of testing your code before you submit it. While you could test cases by hand, using a script to batch-test is a much smarter method.

Usage

I suggest you clone this project to use it, and use pulls later to update.

Config

Before you use the script, you'll need to setup a config.json file. Here's what that would look like:

{
	"languages": {
		"Java 8": { // the name of the language
			"aliases": ["java", "j8"], // how you'll access the language
			"template": "templates/template.java", // template file; see below
			"file": "%s.java", // what the file should be called
			"compile": "javac %s -d %s", // the compile statement (if applicable)
			"run": "java %s" // the run statement
		},
		"Python 3": {
			"aliases": ["python", "py"],
			"template": "templates/template.py",
			"file": "%s.py",
			"run": "python3 %s"
		}
	},
	"directories": {
		"root": "/Programming/Competitions", // path to root directory for contests
		"contest": "%s", // what to name a contest directory
		"src": "src", // what to name a source directory
		"bin": "bin" // what to name a binary directory
	},
	"defaults": {
		"compile_timeout": 15 // how long to wait for compilation
	}
}

Templates

You can also include templates for your source files. To use them, set a "template" property on the language object in the config file with the path to your template relative to the script itself (or just use an absolute path).

Install Packages

Some pip packages are required to run the testing script. To install them, use

pip install -r requirements.txt

Usage

To start the script, use

python tester.py

Please note that only python 3.x is supported at the moment.

Once the script is started, you will be prompted to enter a contest name. This name will correspond to the directory in which all of the contest files are stored. (I personally recommend something like 315D1 for Codeforces #315, Division 1, for example, but it's really up to you.)

After you've entered a contest name, you're presented with a command line. You can type "help" for a full list of commands from there, or I have listed them below:

• add problem lang - Creates new problem files for the given problem in the given language. This will add three files: the source file itself, a test cases input file, and a test cases output file.

• test problem lang - Tests a solution using the test cases in the test cases input file and checking against the answers in the test cases output file.

batch-tester's Issues

Add time and memory limits to problem descriptor

Duh.

Add File I/O Option

ACM often uses file I/O, so including that as an option would definitely be nice.

Add Submit Function

This would be REALLY GOOD.

Probably would require a URL to which to submit. This could be stored in problem descriptors (see #4).

Allow up-arrow to use previous input

It would make testing things many times much easier.

Test case does not advance on TLE

Which means that after a TLE, nothing gets checked against the right output.

Pretty-Print Descriptor

If it's nicely formatted, it's easier to edit.

Add wildcard of some kind

It's often helpful to put in TLE-testing cases without actually knowing the correct answer. There should be a way to do this without it getting marked WA.

Clean up argparse

Including the help message and the dysfunctional quit command.

Empty lines are graded improperly

Title.

Add input parsing

Probably from URL, specifically for CF in this case.

Read all cases before grading

This is pretty logical, since then it's a string comparison instead of an array of string comparison.

Add memory usage

It was here at one time, but I will say that it was really wonky before. Python seems to have several ways of handling this, but none of them seem to be exactly what's needed here.

Add warnings

For example, half TL or half ML would be reasonable cases in which to alert the user.

Problem Descriptor

Adding problem descriptors would be a good feature, since then there would be easy storage of problem information (I/O style, grading type, etc). Could also be helpful for #3.

Fix Timing

The timer always shows up as 1-2ms... even on TLEs of >3s.

Improve graders

For example, adding all of the standard CF checkers is a reasonable step to take.

Don't exit on incorrect arguments

It's really annoying.