eth-sri / dl2 Goto Github PK

Hello!
When I am trying to set some box constraints on part of input I using DSL, the result is not what I expected to have.
DSL is like the following:

FIND i[10]
S.T. i[5:] in [1, 2],
...
RETURN i

I am expecting that last 5 items of i is restricted in [1, 2], but the constraints don't seem to work in the result.

The similar thing happens when I use mask instead of slice operation.
DSL is like the following:

FIND i[10]
S.T. i[mask] in [1, 2],
...
RETURN i

The mask is like 0000000001(only the last one is true), but in the result, seems like the first element in i is restricted by the box constraint.

I'm wondering if I have some misunderstanding about the effect of DSL above? Any advice would be appreciated!

I tried to reproduce the semisupervised learning task reported in the paper, but failed.
The running code is modified based on run.sh, as can be seen in the following.
python main.py --lr 0.001 --net_type vgg --constraint 'DL2' --epochs 1600 --num_labeled 100 --dataset cifar100 --exp_name dl2_06 --constraint-weight 0.6
The results are

| Test Result   Acc@1: 1.12%
| Test Result   CAcc: 99.71%
| Test Result   GroupAcc: 5.18%

Have I used the wrong experimental settings? What parameter settings I should add in order to reproduce the results?
Thanks for any reply.

In querying/context.py, the part after Load gtsrb variables has the following problems:

1. The test images for GTSRB (querying/data/GTSRB/Final_Test/Images) are not located in class folders, only indicating their class by their filenames. Theses should be moved by running this script:

for t in *ppm; do mkdir -p "${t%.*}"; mv "$t" "${t%.*}"; done 

2. The test set needs an absolute path built like this, replacing

testset = datasets.ImageFolder('../data/GTSRB/Final_Test/Images', transform=transform_test)

with

testset = datasets.ImageFolder(os.path.join(curr_dir, 'data/GTSRB/Final_Test/Images'), transform=transform_test)

When executing the following adapted example code given in querying/README.md

import torch

import dl2lib as dl2
import dl2lib.query as q
from configargparse import ArgumentParser

from querying.models.mnist.main import Net1

my_pytorch_model = Net1()
my_pytorch_model.load_state_dict(torch.load('models/mnist/mnist_1.pt'))


def high_level_query():
    qtext = """FIND i[10]
    S.T. i[0] + i[3] < NN(i[4])"""  # actualy query
    context = dict()  # a python dict that defines models and named constants
    context['NN'] = my_pytorch_model
    parser = ArgumentParser(description='DL2 Querying')
    parser = dl2.add_default_parser_args(parser, query=True)
    args = parser.parse_args()  # additionaly arguments such as timeout etc.
    success, result, time = q.Query(qtext, context=context, args=args).run()

if __name__ == '__main__':
    high_level_query()

the following error occurs:

Traceback (most recent call last):
  File "~/dl2-master/querying/example_query.py", line 36, in <module>
    variant1()
  File "~/dl2-master/querying/example_query.py", line 21, in variant1
    success, result, time = q.Query(qtext, context=context, args=args).run()
  File "~/dl2-master/dl2lib/query/query.py", line 11, in __init__
    self.constraint, self.return_values = Parser(query, self.context, self.args).parse()
  File "~/dl2-master/dl2lib/query/parser.py", line 42, in parse
    return self.generate_find()
  File "~/dl2-master/dl2lib/query/parser.py", line 50, in generate_find
    constraints = self.traverse_constraints(self.parse_tree.find.constraints.constraints)
  File "~/dl2-master/dl2lib/query/parser.py", line 153, in traverse_constraints
    c1 = self.traverse_constraint(constraint.c1)
  File "~/dl2-master/dl2lib/query/parser.py", line 137, in traverse_constraint
    lhs = self.traverse_expression(constraint.lhs)
  File "~/dl2-master/dl2lib/query/parser.py", line 127, in traverse_expression
    return eval(f"lhs {exp.op} rhs")
  File "<string>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'NoneType' and 'Fn'

The model mnist_1.pt was trained using the given training procedure train_models.sh from the querying directory.

The line

success, r, t = q.solve(a < b, return_values=[i])

in querying/README.md contains an error.
TypeError: solve() missing 1 required positional argument: 'args'

It would make sense changing the example to

import dl2lib as dl2
import dl2lib.query as q
from configargparse import ArgumentParser

parser = ArgumentParser(description='DL2 Querying')
parser = dl2.add_default_parser_args(parser, query=True)
args = parser.parse_args() # additionaly arguments such as timeout etc.
NN = q.Model(myPytrochModel)
i = q.Variable('i', (10,))
a = i[0] + i[3]
b = NN(i[4])
success, r, t = q.solve(a < b, return_values=[I], args=args)