CAN fuzzing module about caringcaribou

[kasperkarlsson]

Great! I will try to start working on PR #38 later today.

The input() vs raw_input() issue is due to a backwards incompatible change introduced in Python 3, where the input() function of python 2 was completely removed and the input_raw() function was renamed input(). As a workaround, one can simply override the function for one of the Python versions so that the proper function is called regardless of which Python version is running the script.

Regarding the string operations, I will have to make a more thorough analysis when going through the code. String operations tend to be more expensive than using operators on integers and type casting is expensive in itself. If the amount of type casts required are about the same from both sides (i.e. each value is used the same amount of times as an integer and as a string), the operations part will still need to be taken into consideration.

I will also try to find possible causes of the exception you describe. Feel free to open an issue with a short description of your setup and a full stack trace if the problem remains once the PR has been merged!

Also - thank you once again for your contribution! This is really appreciated :-)

[kasperkarlsson]

@FrostTusk FYI: I am doing the changes in the fuzz branch created yesterday; https://github.com/CaringCaribou/caringcaribou/commits/fuzz

[kasperkarlsson]

Great! The log parts were removed in 4eb2f95

[kasperkarlsson]

This module is now officially included in Caring Caribou 😃 A special thanks goes out to @FrostTusk for making this possible!

#40

As always - feel free to post enhancement ideas, bug reports for this module etc as new issues.

[mjidhage]

There are a lot of fuzzing frameworks available, a substantial part written in python. Either reuse/incorporate or provide an interface for them could be an interesting way forward?

[kasperkarlsson]

@mjidhage Good idea. Let's investigate how to do this the best way! If you have any relevant resources, please post them here.

[mjidhage]

Looked through some fuzzing frameworks, found nothing appropriate. Need to invest more brain-hours in this. Random fuzzing is not a problem. Smart fuzzing is the interesting part - perhaps we should create the feature within CC and only make use of an existing pattern-based fuzz data engine?

Investigate https://github.com/jtpereyda/boofuzz/blob/master/INSTALL.rst
also Scapy and https://github.com/MozillaSecurity/peach

[kasperkarlsson]

@mjidhage As you say, creating the feature within CC might be better. Adding CAN support to existing fuzzing frameworks is a great thing, but simply adding a wrapper script for calling an external fuzzer would not add any value to CC.

Let's keep this one open and think a bit more about it. Implementing a basic (stupid) fuzzer would be a no-brainer, but we might want to do something more useful while we're at it. We could of course start off by implementing a simple "fuzz" module and see what ideas we come up with.

[Aezore]

What about a "per ID bitsmasked data fuzzer"?. Let me explain. You could either fix a specific crafted data like all FFs, or 00s or AA55's, with prandom id's which you could: adjust hold time with the arrow keys, stop or "step in/step back". Or you could do the same with the payload; fix the Id and prnd the data with an optional bitmask, again, with arbitrary hold time, start stop and step in/step back.

[kasperkarlsson]

@Aezore Being able to set a bitmask for which bits to fuzz sounds like a straightforward and useful idea, if I am reading your suggestion correctly. As a first step, this could be passed as command line arguments to the module.

How would the hold times be applied? Sending the same data multiple times during the hold time or simply through a delay between each new fuzzed message?

Finally, the part about using e.g. the arrow keys to control the fuzzer's behavior in real-time is interesting and a bit outside of our normal behavior in CC. This could become a useful fuzzing module feature to verify and explore the results of potentially interesting vectors in real-time.

[Aezore]

I reckon it would be better to set another argv for hold frame repetitions, either once per hold, or as much as they fit inside the hold time (if you set +2 repetitions inside a 500ms time gap they would be evenly spaced in time), as I think it would be convenient when you're testing for a suspected function within a range of arb-ids while expecting some behavior in the car for either "trigger it once" or several times.

For the "arrow" keys I found it useful in the can-utils's cansniffer program where you can add or remove arbs on the fly and change behavior like coloring without reloading the program, so it might be a good feature too.

[kasperkarlsson]

These are very interesting ideas. Thank you for the clarification!

[FrostTusk]

Hello, this is my first time posting here. I'm a CS student interested in automotive security. A few months ago I wrote my own fuzzing module using caring caribou (you can find it here). I implemented several different fuzzing algorithms including a brute force algorithm and a mutation based algorithm similar to the one described above.

When I applied them to several automotive testing environments they were surprisingly effective. Maybe you guys could give me some pointers on ways I can improve my fuzzer and perhaps parts of it could even be used as a base for an "official" caring caribou fuzzing module.

As a side note: the module right now doesn't include seeding and real time bit manipulation.

[kasperkarlsson]

Hi there @FrostTusk and thank you for reaching out! Your module looks very interesting. After a quick read of your code, I believe that your algorithm implementations would be great as a base for an official fuzzing module in Caring Caribou!

Looking at purely technical aspects of the code, it seems like some modifications would make it cleaner. Some examples:

• The argument parsing could utilize built-in functionality in argparse to handle both the logic of parse_args() and handle_args(). For example, the currently optional (but in practice mandatory) -alg option should probably be replaced with subparsers for each supported algorithm.
• File handling should in general be wrapped by a with statement and have proper exception handling
• Opening and closing the log file by passing a filename rather than a file handler to write_directive_to_file() adds unnecessary overhead
• There seem to be many type conversions, where payloads are handled as strings rather than integers
• The current implementation is only compatible with Python 2, due to usage of raw_input()

The import of can_actions also needs to be changed from
from can_actions import CanActions, int_from_str_base
to
from lib.can_actions import CanActions, int_from_str_base
in order to map to CC's structural changes introduced in version 0.2.

Suggestion

I would be happy to help fixing the things mentioned above in order to get your fuzzer into CC. Since your fuzzer is not built on a fork of this project, sending a PR is unfortunately impossible in its current state. The way I see it, there are two possible workarounds:

1. You create a new fork of CaringCaribou, copy only your fuzzing.py to the tool/modules subfolder of this fork, make a commit including this change and send a pull request to CC. No code changes are required before you do this - I can take care of the modifications mentioned above in a separate branch for the PR, before merging it as an official CC module.
2. I simply copy the files from your current project into a new branch, modify them and merge.

I would strongly advice for solution 1. This way, it will be clear that the contribution comes from you. It will automatically add you as a contributor (https://github.com/CaringCaribou/caringcaribou/graphs/contributors) and show you in CC's network graph (https://github.com/CaringCaribou/caringcaribou/network).

How do you feel about this? If it seems good to you, please go ahead and send the PR. I will be ready to take care of the technical issues listed above.

[FrostTusk]

Thanks for the advice! I created a new branch and sent a PR.

I'll also give some clarifications on why I initially used some of the less clean implementations.

• Right now when I change raw_input() to input(), a NameError will be thrown when I for example enter "y" (without quotation marks). I believe this might be due to python2's input() behaving differently than python3's and my system might still use python2 for CaringCaribou.
• I used strings internally because that way writing to a file and reading from a file became very easy due to the internal representation of the packet being the same as the one used during file I/O.
  I then used can_actions as a sort of back-end interface for all of the actual CAN bus interactions, this meant I only needed to convert the string to an integer right before it was used by can_actions.
  I assume changing the strings implementation would move the type conversion to the file I/O instead of the CAN I/O, which will be more efficient?

Also the fuzzer does throw a weird error after a large amount of messages have been sent.
The error description changes depending on the platform used, and what version of python-can is being used. On my raspberry pi for example, the fuzzer will throw "NotImplementedError: Invalid CAN Bus Type - None".

[FrostTusk]

@kasperkarlsson Sounds good! In the meantime I've been testing the fuzzer with the last few caring caribou and python-can updates. It seems the error I described above has become worse, but it doesn't look like it's a problem with the fuzzer itself. I'll wait until the branch is merged to open an issue.

[kasperkarlsson]

@FrostTusk I have seen the same issue a lot. By re-using the same CanActions instance rather that re-instantiating it for every request, the risk of race conditions in internal python-can callback handlers during closing of a can.Bus instance is eliminated. I will make a patch for each respective fuzzing function.

By the way, what is the intended usage of the log array created in the fuzzing functions? I can see it being populated, but the contents do not seem to get used for anything.

[FrostTusk]

@kasperkarlsson That makes a lot of sense, thanks for clearing that up for me.

The log array isn't used right now, I implemented it because I was perhaps going to try and use machine learning on the past X amount of messages (or maybe responses) that would be contained in the log array. Since I didn't end up implementing that, you can certainly just remove the log arrays.

[kasperkarlsson]

All fuzzing modes are working properly, but some minor things are left to look into. Should be ready for merging soon! 😃

[FrostTusk]

Great! I'm also really honored that you guys ended up using my fuzzer as a base for this module.