cleverhans-lab / cleverhans Goto Github PK

Having flags (tf flags) is ok in the tutorials, but it would be easier to import functions from the cleverhans module if they didn't reuse flags directly but rather got these values as arguments (e.g., learning rate in model train)

According to the paper Adversarial examples in the physical world, FGSM uses y_true, the true label, to generate the adversarial images, instead of model predictions. Besides, it does not really make sense to me to use y_pred and logits to compute the loss since y_pred = softmax(logits). Instead of

def fgsm(x, predictions, eps, clip_min=None, clip_max=None)

The second parameter should be

def fgsm(x, y_true, eps, clip_min=None, clip_max=None)

transposing from theano ordering may be adding overhead

Running mnist_tutorial_th.py gives:

Using gpu device 0: GeForce GTX TITAN X (CNMeM is disabled, cuDNN 5103)
Using Theano backend.
X_train shape: (60000, 1, 28, 28)
60000 train samples
10000 test samples
Loaded MNIST test data.
Traceback (most recent call last):
  File "/part/01/Tmp/lisa/os_v5/anaconda/lib/python2.7/site-packages/ipdb/__main__.py", line 164, in main
    pdb._runscript(mainpyfile)
  File "/Tmp/lisa/os_v5/anaconda/lib/python2.7/pdb.py", line 1233, in _runscript
    self.run(statement)
  File "/Tmp/lisa/os_v5/anaconda/lib/python2.7/bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "tutorials/mnist_tutorial_th.py", line 1, in <module>
    from __future__ import absolute_import
  File "tutorials/mnist_tutorial_th.py", line 46, in main
    model = model_mnist()
  File "/u/jastrzes/deep_protein/notebooks/cleverhans/cleverhans/utils_mnist.py", line 66, in model_mnist
    model.add(Dense(nb_classes))
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/models.py", line 331, in add
    output_tensor = layer(self.outputs[0])
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/topology.py", line 543, in __call__
    self.build(input_shapes[0])
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/layers/core.py", line 752, in build
    constraint=self.W_constraint)
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/topology.py", line 415, in add_weight
    weight = initializer(shape, name=name)
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/initializations.py", line 60, in glorot_uniform
    return uniform(shape, s, name=name)
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/initializations.py", line 33, in uniform
    return K.random_uniform_variable(shape, -scale, scale, name=name)
  File "/u/jastrzes/.local/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/backend/theano_backend.py", line 178, in random_uniform_variable
    return variable(np.random.uniform(low=low, high=high, size=shape),
  File "mtrand.pyx", line 1252, in mtrand.RandomState.uniform (numpy/random/mtrand/mtrand.c:13528)
OverflowError: Range exceeds valid bounds

This happens because default dim ordering in Keras is now "tf", which means that `Convolution2D in https://github.com/openai/cleverhans/blob/master/cleverhans/utils_mnist.py#L53 interprets "1" as the width of image

The graph could be ran with tf.set_random_seed() for reproducibility

I'm getting 3.7 seconds per 100 batches.
In "Explaining and Harnessing", I think it took something like 4 seconds per full epoch, with older hardware.
Maybe the model in the tutorial is just a lot bigger.

Create an iterator so one can write for batch_index in dataset

the saliency map attack reports the average distortion for all samples, would be nice to print a different number for samples that were successfully misclassified

I run the test_mnist_accuracy.py, meet the following error:
Using TensorFlow backend.
X_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
/anaconda/lib/python2.7/site-packages/cleverhans-1.0.0-py2.7.egg/cleverhans/utils.py:143: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(64, (8, 8), padding="same", strides=(2, 2))
border_mode="same"
/anaconda/lib/python2.7/site-packages/cleverhans-1.0.0-py2.7.egg/cleverhans/utils.py:147: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(128, (6, 6), padding="valid", strides=(2, 2))
border_mode="valid"),
/anaconda/lib/python2.7/site-packages/cleverhans-1.0.0-py2.7.egg/cleverhans/utils.py:149: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(128, (5, 5), strides=(1, 1))
Convolution2D(nb_filters * 2, 5, 5, subsample=(1, 1)),
Defined TensorFlow model graph.
Epoch 0
Epoch took 8.60225486755 seconds
Epoch 1
Epoch took 8.84567499161 seconds
Completed model training.
Traceback (most recent call last):

File "", line 1, in
runfile('/Users/adhockuang/myworkspace/cleverhans-master/tests_tf/test_mnist_accuracy.py', wdir='/Users/adhockuang/myworkspace/cleverhans-master/tests_tf')

File "/anaconda/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile
execfile(filename, namespace)

File "/anaconda/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile
builtins.execfile(filename, *where)

File "/Users/adhockuang/myworkspace/cleverhans-master/tests_tf/test_mnist_accuracy.py", line 84, in
app.run()

File "/anaconda/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))

File "/Users/adhockuang/myworkspace/cleverhans-master/tests_tf/test_mnist_accuracy.py", line 76, in main
args=eval_params)

File "/anaconda/lib/python2.7/site-packages/cleverhans-1.0.0-py2.7.egg/cleverhans/utils_tf.py", line 206, in model_eval
keras.backend.learning_phase(): 0})

ValueError: operands could not be broadcast together with shapes (128,) (80,) (128,)

IMHO, there might be a couple of improvements over the JSMA API.

1. increase and theta are duplicates.
   increase=True <-> theta=+x
increase=False <-> theta=-x
2. jacobian_graph is like a loose ends to me.
   I think the gradients should be calculated internally in the JSMA function, it is, at least to me, counter-intuitive to called some random function before calling the jsma.

I got something > .6. Not sure why this would be; I'd expect something like .99 for such a large eps.

also, in particular, confirm that eps of 0.3 in the tutorial results in images that are easy for humans to read. this is higher than i've used before

python tutorials/mnist_tutorial_th.py Using TensorFlow backend. I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally X_train shape: (60000, 1, 28, 28) 60000 train samples 10000 test samples Loaded MNIST test data. Traceback (most recent call last): File "tutorials/mnist_tutorial_th.py", line 98, in <module> main() File "tutorials/mnist_tutorial_th.py", line 48, in main predictions = model(x) File "/home/ian/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 574, in __call__ outputs = to_list(self.call(x, mask)) File "/home/ian/.local/lib/python3.5/site-packages/keras/models.py", line 381, in call return self.model.call(x, mask) File "/home/ian/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 2240, in call output_tensors, output_masks, output_shapes = self.run_internal_graph(inputs, masks) File "/home/ian/.local/lib/python3.5/site-packages/keras/engine/topology.py", line 2383, in run_internal_graph computed_mask)) File "/home/ian/.local/lib/python3.5/site-packages/keras/layers/core.py", line 90, in call x = K.in_train_phase(K.dropout(x, self.p, noise_shape), x) File "/home/ian/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2051, in dropout return tf.nn.dropout(x * 1., retain_prob, noise_shape, seed=seed) File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/ops/nn_ops.py", line 1743, in dropout x = ops.convert_to_tensor(x, name="x") File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 669, in convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function return constant(v, dtype=dtype, name=name) File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/framework/constant_op.py", line 165, in constant tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape)) File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in make_tensor_proto tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values]) File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/framework/tensor_util.py", line 441, in <listcomp> tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values]) File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/util/compat.py", line 65, in as_bytes (bytes_or_text,)) TypeError: Expected binary or unicode string, got Elemwise{mul,no_inplace}.0

following changes made by #79

How can I save only the images generated from either FGSM or JSMA attacks?

It looks like the tensor of images returned out of batch_eval(sess, [x], [adv_x], [X_train], args=eval_params) only give the noise that will be added to the original images.

I'm training a detector, and this library looks much better than the way I was generating fooling images before. I just can't figure out how to get the actual perturbed images.

Thanks.

This seems harmless, but it is annoying. Not sure if we can fix it, since it may be a bug in TensorFlow itself.

At the end of the MNIST tutorial, I get this error message:
Exception AttributeError: AttributeError("'NoneType' object has no attribute 'raise_exception_on_not_ok_status'",) in <bound method Session.del of <tensorflow.python.client.session.Session object at 0x7fb09c05ad50>> ignored

tf implementation seems somewhat slower than theano code written 2 years ago

should benchmark it and make sure the runtime is correct.

Traceback (most recent call last):
File "tutorials/mnist_tutorial_jsma.py", line 140, in
app.run()
File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 43, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "tutorials/mnist_tutorial_jsma.py", line 83, in main
saver.save(sess, save_path)
File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1329, in save
self.last_checkpoints, latest_filename)
File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 745, in update_checkpoint_state
text_format.MessageToString(ckpt))
File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/lib/io/file_io.py", line 351, in atomic_write_string_to_file
rename(temp_pathname, filename, overwrite=True)
File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/lib/io/file_io.py", line 333, in rename
compat.as_bytes(oldname), compat.as_bytes(newname), overwrite, status)
File "/opt/anaconda/4.2.0/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/ian/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.PermissionDeniedError: /tmp/checkpoint.tmp87146825c2f043aa9c3fd9f92e3efa98

in JSMA

batch_eval in utils_th (https://github.com/openai/cleverhans/blob/master/cleverhans/utils_th.py#L167) throws an error if computation graph doesn't use learning_phase. Way to fix that would be to add on_unused_input='ignore' to arguments of theano.function call.

right now the map definitions are biased towards image data (because of row x col). We should clean up the code to define them for any input vector. We can do that once the tutorial is fully merged (branch saliency is closed)

with current model, after 5 epochs of adversarial training, clean accuracy should definitely be .97 or better, adv accuracy should definitely be .79 or better. usually better than this but things aren't seeded so it fluctuates

In utils_tf.py, line 26 and 29, it should be categorical_crossentropy(model, y) instead of categorical_crossentropy(y, model). This won't affect training in a noticeable way for mnist but will make the training fail to work when I tried a deeper model on cifar-10.

I get this when I run the unmodified version of the tutorial on CPU, with the latest keras.
The error does not show up with keras 1.2

(tensorflow) rva5120@cse-p336mcro05:~/tensorflow/cleverhans/tutorials$ python mnist_tutorial_tf.py
Using TensorFlow backend.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
X_train shape: (60000, 28, 28, 1)
60000 train samples
10000 test samples
/home/grads/rva5120/tensorflow/cleverhans/cleverhans/utils.py:143: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(64, (8, 8), padding="same", strides=(2, 2))`
  border_mode="same"
/home/grads/rva5120/tensorflow/cleverhans/cleverhans/utils.py:147: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(128, (6, 6), padding="valid", strides=(2, 2))`
  border_mode="valid"),
/home/grads/rva5120/tensorflow/cleverhans/cleverhans/utils.py:149: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(128, (5, 5), strides=(1, 1))`
  Convolution2D(nb_filters * 2, 5, 5, subsample=(1, 1)),
Defined TensorFlow model graph.
Epoch 0
	Epoch took 52.5618000031 seconds
Traceback (most recent call last):
  File "mnist_tutorial_tf.py", line 122, in <module>
    app.run()
  File "/home/grads/rva5120/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "mnist_tutorial_tf.py", line 81, in main
    evaluate=evaluate, args=train_params)
  File "/home/grads/rva5120/tensorflow/cleverhans/cleverhans/utils_tf.py", line 142, in model_train
    evaluate()
  File "mnist_tutorial_tf.py", line 70, in evaluate
    args=eval_params)
  File "/home/grads/rva5120/tensorflow/cleverhans/cleverhans/utils_tf.py", line 206, in model_eval
    keras.backend.learning_phase(): 0})
ValueError: operands could not be broadcast together with shapes (128,) (16,) (128,) 

Improve JSMA implementation so we can feed it more than one input at a time

note to self: two empty lines before each function

this will be helpful when working on source-target misclassification

This tutorial is still using the TH ordering, would be nice to change it to TF

Completed model training.
Traceback (most recent call last):
File "test_mnist_accuracy.py", line 71, in
app.run()
File "/home/mahnerak/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 44, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "test_mnist_accuracy.py", line 67, in main
assert float(accuracy) >= 0.98, accuracy
AssertionError: 0.9639

Hello @goodfeli and @npapernot ,

In my understanding, these two lines get the result of what trained model is gonna classify the clean example.

Then by adding this gradient to the image, the trained model will tend to not classify it as the same label as before.

However, I'm wondering how can I use FSGM to fool model so that it will classify all examples to a specified label, e.g., 0.

vim is showing me red marks on indentations in test_mnist_accuracy.py. We should set up a standard way to configure our editors.

We should do a new release with a major version increment because of #25 . Because the loss was used as part of the attacks, this bug made the attacks appear too weak.

I was using the JSMA attack (with the increase=False option) on "tutorial" MNIST classifier from the TensorFlow website. However, it occasionally dies when trying to remove a key that does not exist:

Current iteration: 5 - Current Prediction: 1
Current iteration: 10 - Current Prediction: 1
Current iteration: 15 - Current Prediction: 1
Current iteration: 20 - Current Prediction: 1
Current iteration: 25 - Current Prediction: 3
Traceback (most recent call last):
  File "mnist_test_updated.py", line 245, in <module>
    max_distortion/100, False, 0, 1)
  File "/net/af11/apn4za/AdversarialMachineLearning/fjsma/cleverhans/cleverhans/attacks_tf.py", line 246, in jsma_tf
    grads_target, grads_others, search_domain, increase)
  File "/net/af11/apn4za/AdversarialMachineLearning/fjsma/cleverhans/cleverhans/attacks_tf.py", line 124, in saliency_map
    search_domain.remove(p1)
KeyError: 0

Link to offending lines.

This occurs when there are no pixels that satisfy the saliency map condition of alpha < 0 and beta > 0 (using the terms from Papernot, et al), so then "best" is 0. However, for MNIST, the 0 pixel is rarely set, so it is not in the search_domain set and the removal fails.

I did a temporary fix on my side by using discard instead of remove, which fails silently. The downside of this is that the saliency map will continue to delete the 0, 0 pixels until the max_iters value is exceeded in the main jsma_tf loop; this wastes resources, but is a simple fix.

ideally we would only need keras for the tutorial, so that people using other frameworks can run the benchmarks

also might be worth renaming into cnn_model since it is not specific to mnist in anyway

was i meant to need to set image order to 'th' in ~/.keras?

For now, I guess we have to run each of them manually? We should either copy the docker setup from OpenAI Gym or the nose setup from Pylearn2.

from keras.backend import categorical_crossentropy

check if needed and remove