emerge-lab / regression-testing

Regression testing tool using Wandb API.

Usage:

The tool can be used through importing the PyPI package for the tool:

from regressiontestingreport import report_generation

From here, a report can be generated by calling the "generate_report" function with a parameter you want to track, the name of your Wandb project project, and the name of your Wandb entity:

report_generation.generate_report(<parameter_name>, <project_name>, <entity_name>)

Your Wandb key should be included as a environment variable if you wish to automatically log in (Wandb may prompt you to create an account otherwise).

Example Usage:

The report generation function can be used in tandem with arg-parsing and a job scheduler to automatically generate a report every day. An example of a simple script that takes in inputs for the report function through the command line and then generates a report with them is provided below:

daily_report.py:

# Imports
from regressiontestingreport import report_generation
import wandb
import argparse

# Login
wandb.login(key=wandb_key)


# Arg-parsing to take in <parameter_name>, <project_name>, <entity_name> from the command line
parser = argparse.ArgumentParser(description='Take in inputs')
parser.add_argument('my_param', metavar='param', type=str, nargs=1,
                    help='The parameter you wish to log')
parser.add_argument('project_name', metavar='project name', type=str, nargs=1,
                    help='The name of the project')
parser.add_argument('entity_name', metavar='entity name', type=str, nargs=1,
                    help='The name of the entity who owns the project')
args = parser.parse_args()

# Generate and return a report 
my_report = report_generation.generate_report(args.my_param[0], args.project_name[0], args.entity_name[0])

Daily Report Generation Through Cronjob:

Cron is a tool that can help schedule jobs on Unix-like operating systems. The syntax generally involves 5 arguments that specify the schedule (min [0-59], hour [0-23], day of month [1-31], month [1-12], and day of week [0-6]), followed by a command to execute. An example for daily report generation at 2:00 AM is provided below:

0 2 * * * <path_to_python> <path_to_python_script> <parameter_name> <project_name> <entity_name>