yenchenlin / rl-attack-detection

Code for "Detecting Adversarial Attacks on Neural Network Policies with Visual Foresight"

Detecting Adversarial Attacks on Neural Network Policies with Visual Foresight

DISCLAIMER: This repository is a modified version of openai/baselines.

Publication

Paper: https://drive.google.com/file/d/0B50cbskLVq-ed2F3eUw4SWQxbUU/view

@article{Lin2017RLAttackDetection,
  title={Detecting Adversarial Attacks on Neural Network Policies with Visual Foresight},
  author={Lin, Yen-Chen and Liu, Ming-Yu and Sun, Min and Huang, Jia-Bin},
  journal={arXiv preprint arXiv:1710.00814},
  year={2017}
}

Dependencies

• Python 3
• cleverhans v2.0.0

pip install -e git+http://github.com/tensorflow/cleverhans.git#egg=cleverhans

• others (e.g., gym, baselines, ...)

git clone https://github.com/yenchenlin/rl-attack-detection.git
cd rl-attack-detection
pip install -e .

Example

Here I'll use Atari game Freeway as an example to demonstrate how to run the code.

Let's start by switch to the home directory:

cd rl-attack-detection

1. Download pre-trained agent

Download this repository which contains pre-trained DQN agents for Freeway to ./atari-pre-trained-agents/.

2. Run pre-trained agent

Test the performance of the pre-trained agent:

python -m baselines.deepq.experiments.atari.enjoy --model-dir ./atari-pre-trained-agents/Freeway --env Freeway

For game Freeway, you should see output similar to follows:

29.0
27.0
28.0
...

This means that our agent is now a master of the game!

3. Perform adversarial attack

Use adversarial example crafted by FGSM to attack deep RL agent:

python -m baselines.deepq.experiments.atari.enjoy --model-dir ./atari-pre-trained-agents/Freeway --env Freeway --attack fgsm

Other attacks: argument passed to --attack can be fgsm, iterative, cwl2.

You should see output similar to follows:

0.0
0.0
0.0
...

which means that the agent is fooled by adversary and went crazy!

4. Use visual foresight as defense

To protect the agent, first download this repository which contains pre-trained visual foresight module for Freeway to ./atari-visual-foresight/.

Then, we can use visual foresight to protect deep RL agent:

python -m baselines.deepq.experiments.atari.enjoy --model-dir ./atari-pre-trained-agents/Freeway --env Freeway --attack fgsm --defense foresight

Now, you should see similar outputs to step. 2, which means that our agents work well again.

Add More Attacks

To use new attack methods, you can add the attack code here. Generally, attack methods that follow the interface of cleverhans can be added within few lines.

rl-attack-detection's Issues

ModuleNotFoundError: No module named 'tensorflow_addons'

Hi, I got this issue when running the code:

File "/home/ubuntu/src/cleverhans/cleverhans/attacks/init.py", line 27, in
from cleverhans.attacks.spsa import SPSA, projected_optimization
File "/home/ubuntu/src/cleverhans/cleverhans/attacks/spsa.py", line 9, in
import tensorflow_addons as tfa
ModuleNotFoundError: No module named 'tensorflow_addons'

However, tensorflow_addons is only for tf2, but your codes are based on tf 1. How could I solve this problem?

Gym Monitoring

How did you train your agent?

May I ask how did you train your agent? I used the standard baselines method:
python -m baselines.run --alg=deepq --env=PongNoFrameskip-v4 --num_timesteps=1e7 --save_path=~/models/pong_10M_deepq
However, the trained model cannot be used due to the following issue. I believe it is due to differrent parameters you set to train your agent and the default baselines setting.

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1292, in _do_call
return fn(*args)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1277, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1367, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [512] rhs shape= [256]
[[{{node deepq/save/Assign_1}} = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deepq/q_func/action_value/fully_connected/biases, deepq/save/RestoreV2:1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1538, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 887, in run
run_metadata_ptr)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1110, in _run
feed_dict_tensor, options, run_metadata)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1286, in _do_run
run_metadata)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1308, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [512] rhs shape= [256]
[[{{node deepq/save/Assign_1}} = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deepq/q_func/action_value/fully_connected/biases, deepq/save/RestoreV2:1)]]

Caused by op 'deepq/save/Assign_1', defined at:
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/deepq/experiments/atari/enjoy.py", line 145, in
model_path=os.path.join(args.model_dir, "saved")
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/deepq/build_graph.py", line 130, in build_act
U.load_state(model_path)
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/common/tf_util.py", line 272, in load_state
saver = tf.train.Saver()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1094, in init
self.build()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1106, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1143, in _build
build_save=build_save, build_restore=build_restore)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 787, in _build_internal
restore_sequentially, reshape)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 119, in restore
self.op.get_shape().is_fully_defined())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
validate_shape=validate_shape)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3272, in create_op
op_def=op_def)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1768, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [512] rhs shape= [256]
[[{{node deepq/save/Assign_1}} = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deepq/q_func/action_value/fully_connected/biases, deepq/save/RestoreV2:1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/deepq/experiments/atari/enjoy.py", line 145, in
model_path=os.path.join(args.model_dir, "saved")
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/deepq/build_graph.py", line 130, in build_act
U.load_state(model_path)
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/common/tf_util.py", line 273, in load_state
saver.restore(get_session(), fname)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1574, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [512] rhs shape= [256]
[[{{node deepq/save/Assign_1}} = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deepq/q_func/action_value/fully_connected/biases, deepq/save/RestoreV2:1)]]

Caused by op 'deepq/save/Assign_1', defined at:
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/deepq/experiments/atari/enjoy.py", line 145, in
model_path=os.path.join(args.model_dir, "saved")
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/deepq/build_graph.py", line 130, in build_act
U.load_state(model_path)
File "/Users/harry/Documents/FYP_Codes/rl-attack-detection/baselines/common/tf_util.py", line 272, in load_state
saver = tf.train.Saver()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1094, in init
self.build()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1106, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1143, in _build
build_save=build_save, build_restore=build_restore)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 787, in _build_internal
restore_sequentially, reshape)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 119, in restore
self.op.get_shape().is_fully_defined())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
validate_shape=validate_shape)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3272, in create_op
op_def=op_def)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1768, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [512] rhs shape= [256]
[[{{node deepq/save/Assign_1}} = Assign[T=DT_FLOAT, _class=["loc:@deepq/q_func/action_value/fully_connected/biases"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](deepq/q_func/action_value/fully_connected/biases, deepq/save/RestoreV2:1)]]

Demo GIF