udopia / gbd Goto Github PK

View Code? Open in Web Editor NEW

14.0 1.0 4.0 47.18 MB

Management of Benchmark Instances and Instance Attributes

Home Page: https://udopia.github.io/gbd/

License: MIT License

Python 87.73% Shell 0.55% HTML 3.86% CSS 1.05% JavaScript 6.81%

gbd's Introduction

GBD Benchmark Database

GBD Benchmark Database (GBD) contributes data to your solver evaluations.

GBD has three interfaces

Command-line interface gbd
Web interface gbd serve
Python interface gbd_core.api.GBD

GBD solves several problems

benchmark instance identification and identification of equivalence classes of benchmark instances (gbdhash, isohash, instance family, ...)
distribution of benchmark instances and benchmark meta-data
a simple query language to provide access to filtered sets of benchmark instances and instance features
initialization and maintenance of instance feature databases (meta.db, base.db, gate.db, ...)
transformation algorithms for benchmark instances such as instance sanitization or transformation of cnf to k-isp instances
keeping track of contexts (cnf, sanitized-cnf, k-isp, ...) and relations of instances between contexts

Programming Language

Python 3
SQLite

Installation

pip install gbd-tools

Configuration

fetch a database, e.g., https://benchmark-database.de/getdatabase/meta.db
export GBD_DB=[path/to/database1]:[path/to/database2:..] (and put it in your .bashrc)
test command-line interface with commands gbd info and gbd --help

GBD Python Interface

from gbd_core.api import GBD
with GBD(['path/to/database1', 'path/to/database2', ..] as gbd:
    df = gbd.query("family = hardware-bmc", resolve=['verified-result', 'runtime-kissat'])

GBD Server

This runs under https://benchmark-database.de/. The command is available in gbd-tools: gbd serve --help

GBD Command-Line Interface

gbd get

We assume https://benchmark-database.de/getdatabase/meta.db is in your gbd path GBD_DB.

Get list of benchmark instances in database:

gbd get

Get list of benchmark instances including some meta-data

gbd get -r result family

Filter for specific benchmark instances with gbd-query

gbd get "family = hardware-bmc" -r filename

gbd init

We assume you installed the python extension module gdbc.

All initialization routines can run in parallel and per-process ressource limits can be set. See gbd init --help for more info.

gbd init local

To initialize a database with local paths to your own benchmarks:

gbd -d my.db init local [path/to/benchmarks]

After that in my.db, the features local and filename exist and are associated with their corresponding gbd-hash:

gbd -d my.db get -r local filename

gbd init isohash

To identify isomorphic instances (approximately by the hash of the sorted degree-sequence of their graph representation):

gbd -d my.db init isohash

After that in my.db, instances can be grouped by their isohash:

gbd -d my.db get -r local filename -g isohash

gbd init base, gbd init gate

gbd -d my.db:mybase.db init --target mybase base

gbd -d my.db:mygate.db init --target mygate gate

gbd's People

Contributors

Stargazers

Watchers

Forkers

weitspringer robbepop infolisa chrjabs

gbd's Issues

First database feature claimed to be `None`

GBD version: 4.1.2 (current one)

When queried for the feature names in a database, GBD seems to return one more element than there are actual features. This element is None and appears first in the list:

import gbd_tool.gbd_api

with gbd_tool.gbd_api.GBD(dbs=['base.db']) as api:
    features = api.get_features()
    print(features[0] is None)

This happens for all database files I tested (even older ones where I'm sure that they don't contain a column whose name could be interpreted as None).

`get_features()` only returns feature names of one database

... if called with dbname=None (the default), but gbd_tool.gbd_api.GBD() initialized with a string representing multiple databases (separated by :).

-t parameter not recognized

When executing the command "gbd eval par2 "competition_track = main_2020" -t 5000 -r kissat_sat relaxed_newtech" (copied from the "Collaborative Management of Benchmark Instances and their Attributes" PDF). The following error occurs:

usage: gbd [-h] [-d [DB]] [-j [JOBS]] [-t TIMEOUT] [-s { ,,,;}] [--join-type {INNER,OUTER,LEFT}] [-v] {init,hash,get,set,import,create,delete,rename,info,info_set,info_clear,eval,plot,graph} ... gbd: error: unrecognized arguments: -t 5000

Removing the "-t 5000" from the command the terminal returns the expected result:

3926.19 264/400 kissat_sat 4140.31 255/400 relaxed_newtech 3323.5 285/400 VBS

This issue could be reproduced on Linux Mint 20 Cinnamon (4.6.7), a Virtual Machine running Linux Mint 20.2 Cinnamon (5.0.4) and a Windows Subsystem running Ubuntu 20.04 LTS

Import Error for gbd eval comb

When using the command "gbd eval comb "competition_track = main_2020" -k 3 -r cadical_sc2020 kissat exmaple_padc_dl ... | sort" (note: removed -t 5000, because of #17 ) the follwing error occurs:

Traceback (most recent call last): File "/home/u1/.local/bin/gbd", line 8, in <module> sys.exit(main()) File "/home/u1/.local/bin/gbd.py", line 315, in main args.func(api, args) File "/home/u1/.local/bin/gbd.py", line 103, in cli_eval_combinations from gbd_tool import eval_comp_ilp as eci ImportError: cannot import name 'eval_comp_ilp' from 'gbd_tool' (/home/u1/.local/lib/python3.8/site-packages/gbd_tool/__init__.py)