netzob / netzob

Netzob: Protocol Reverse Engineering, Modeling and Fuzzing

Netzob : Protocol Reverse Engineering, Modeling and Fuzzing

See complete documentation here: https://netzob.github.io/netzob/

About Netzob

Netzob is an open source tool for reverse engineering, modelization, traffic generation and fuzzing of communication protocols.

Netzob is suitable for reversing network protocols, structured files and system and process flows (IPC and communication with drivers and devices). Netzob handles different types of protocols: text protocols (like HTTP and IRC), delimiter-based protocols, fixed fields protocols (like IP and TCP) and variable-length fields protocols (like TLV-based protocols).

Netzob can be used to infer the message format and the state machine of a protocol through passive and active processes. Its objective is to bring state of art academic researches to the operational field, by leveraging bio-informatic and grammatical inferring algorithms in a semi-automatic manner.

Once modeled or inferred, a protocol model can be used in our traffic generation engine, to allow simulation of realistic and controllable communication endpoints and flows.

Main features of Netzob

The main features of Netzob are:

Protocol Modelization

Netzob includes a complete model to represent the message format (aka its vocabulary) and the state machine of a protocol (aka its grammar).

Protocol Inference

The vocabulary and grammar inference component provides both passive and active reverse engineering of communication flows through automated and manuals mechanisms.

Traffic Generation

Given vocabulary and grammar models previously inferred or modelized, Netzob can understand and generate communication traffic with remote peers. It can thus act as either a client, a server or both.

Protocol Fuzzing

Netzob helps security evaluators by simplifying the creation of fuzzers for proprietary or undocumented protocols. Netzob considers the format message and state machine of the protocol to generate optimized and specific test cases. Both mutation and generation are available for fuzzing.

Import Communication Traces

Data import is available in two ways: either by leveraging the channel-specific captors (currently network and IPC -- Inter-Process Communication), or by using specific importers (such as PCAP files, structured files and OSpy files).

Export Protocol Models

This module permits to export an model of a protocol in formats that are understandable by third party software or by a human. Current work focuses on export format compatible with main traffic dissectors (Wireshark and Scapy) and fuzzers (Peach and Sulley).

Netzob must be used as a Python 3 library. It can either be imported in your scripts or in your favorite interactive shell (ipython?).

More Information

Website:	https://github.com/netzob/netzob
Twitter:	Follow Netzob's official accounts (@Netzob)

Netzob has been initiated by security auditors of AMOSSYS and the CIDre research team of CentraleSupélec to address the reverse engineering and fuzzing of communication protocols.

Documentation

The documentation is available online at: https://netzob.github.io/netzob/

If you want to build the documentation, run the following command:

$ sphinx-build -b html doc/documentation/source/ doc/documentation/build/

Get Started with Netzob

Install it

Installing Netzob system dependencies

First thing to do is to check the version of your python3 interpretor. Netzob requires at least Python 3.8:

$ python3 --version
Python 3.8.10

You have to install the following system dependencies:

$ apt-get install -y python3 python3-dev python3-setuptools virtualenv build-essential libpcap-dev libgraph-easy-perl libffi-dev

Then, create a virtualenv:

$ mkdir venv
$ virtualenv venv
$ source venv/bin/activate

Installing Netzob from PyPI

You can install Netzob from PyPI (recommended choice):

(venv) $ pip3 install netzob

Installing Netzob from sources

If you have retrieved Netzob sources, the installation procedure is:

(venv) $ pip3 install Cython==0.29.32  # Should be manually installed because of setup.py direct dependency
(venv) $ pip3 install -e .

API usage

Once installed, we recommend to use the Netzob API inside scripts, with the following statement to import Netzob:

from netzob.all import *

Start Netzob CLI

Netzob also provides its own CLI, in order to play interactively with it:

(venv) $ netzob

Contributing

There are multiple ways to help-us.

Defects and Features Requests

Help-us by reporting bugs and requesting features using the Bug Tracker.

Join the Development Team

To participate in the development, you need to get the latest version, modify it and submit your changes.

You're interested in joining, please contact us!

Authors, Contributors and Sponsors

See the top distribution file AUTHORS.txt for the detailed and updated list of authors, contributors and sponsors.

Licenses

This software is provided under the GPLv3 License. See the COPYING.txt file in the top distribution directory for the full license text.

The documentation is under the CC-BY-SA licence.

Extra

Zoby, the official mascot of Netzob.

netzob's Issues

PCAPImporter's definition of network layers is shifted by one

The generation of layered message objects erroneously shifts the payload compared to the correct definition by an offset of one network layer in PCAPImporter.__packetHandler() for the arguments

• layer=2 : L2NetworkMessage
• layer=3 : L3NetworkMessage and
• layer=4 : L4NetworkMessage

GUI doesn't start

./netzob doesn't start the GUI but gives the python interpreter. Opening that file the following lines are commented.

Launch the GUI or the plugin manager

# if commandLineParser.isStartGUIRequested():
#     GObject.threads_init()  # for handling concurrent GUI access from threads
#     from netzob.NetzobMainController import NetzobMainController
#     netzobController = NetzobMainController()
#     netzobController.run()
# # Display the management of plugins
# elif commandLineParser.isManagePluginsRequested():
#     from netzob.NetzobPluginManagement import NetzobPluginManagement
#     netzobPluginManagement = NetzobPluginManagement(commandLineParser)
#     netzobPluginManagement.start()
# # Launch the interactive session
# elif commandLineParser.isInteractiveConsoleRequested():
#     from netzob.NetzobInteractiveSessionController import NetzobInteractiveSessionController
#     interactiveSession = NetzobInteractiveSessionController()
#     interactiveSession.start()

Removing the # leads to some import errors.
How to start the GUI?

How to export to wirshark using the API?

Since there is no GUI :(

Failed to install from github sources

Python version:
3.6.2

Commands:

git checkout [netzob-url]
sudo python3 setup.py install
netzob

Error msg:

Traceback (most recent call last):
  File "/usr/bin/netzob", line 4, in <module>
    __import__('pkg_resources').run_script('Netzob==1.0.2', 'netzob')
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 741, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1494, in run_script
    raise ResolutionError("No script named %r" % script_name)
pkg_resources.ResolutionError: No script named 'netzob'

The pip version works.
Greetings

Implements DBSCAN clustering

We could give a try with the DBSCAN clustering algorithm.

https://gitlab.com/SyBen/dbscan/blob/master/launcher.py

Move sign, endianness and unit size attributes from AbstractType to Integer

As those attributes are generally more related to Integer encoding, it would probably be better to move them from AbstractType to Integer, in order to reduce from AbstractType complexity.

Some use cases to support:

• integer with endianness, sign and unit size
• IPv4 that can be encoded in little or big endian

Adding support for user specified automata?

As far as I know, the current implementation of automata learning in netzob is quite primitive, where only three types of automata are supported, i.e., chained states automata, one state automata, and PTA automata. This will not be enough for real world cases.

I understand that it is not so easy to learn accurate automata from PCAP files. However, do you guys think it would be better if we give the users an API to specify/modify the automata?

AFAIK, many protocol Fuzzers, such as Sulley and Peach, allow users to specify automata.

[Feature Request] Bluetooth Serial Port interface/Universal Serial Bus interface protocol analysis.

I have a bunch of uses for this tool for analyzing serial protocols in use, such as over a normal serial port, Universal Serial Port (USB) or Bluetooth SPP/Generic profile.

Preferably usable on Windows for USB and Android for Bluetooth.

Netzob crashes while importing PCAP files

I tried to import a PCAP file of size 8 mb while following the tutorial on the main website. Netzob has crashed multiple times. The following is the traceback for that exception. Not being able to import PCAPs is a huge inconvenience.
`DEBUG:root: Local configuration file used: /root/.netzob
DEBUG:root: Workspace path declared in configuration file: /root/workspace/netzob
DEBUG:root: Workspace R/W access is valid: /root/workspace/netzob
DEBUG:root:+ Load workspace...
DEBUG:root: Workspace configuration file found: /root/workspace/netzob/workspace.xml
DEBUG:root: Workspace configuration file /root/workspace/netzob/workspace.xml is valid against XSD scheme /usr/share/netzob/xsds/0.1/Workspace.xsd
DEBUG:root:Bug reporter disabled.
DEBUG:root:Logging config file: /root/workspace/netzob/logging/logging.conf
[MainThread]2016-03-21 09:52:00,756 - NetzobMainController - INFO - Starting netzob UI
[MainThread]2016-03-21 09:52:22,745 - Project - INFO - Save the config file of project monday 21st in /root/workspace/netzob/projects/751429ce-fa7e-4403-93ef-f96322ed7cc1/config.xml
[MainThread]2016-03-21 09:52:22,746 - Project - INFO - Creation of the directory: /root/workspace/netzob/projects/751429ce-fa7e-4403-93ef-f96322ed7cc1
[MainThread]2016-03-21 09:52:22,746 - Workspace - INFO - Save the config file of the workspace New Workspace in /root/workspace/netzob/workspace.xml
[MainThread]2016-03-21 09:52:23,432 - Project - INFO - Loading project 'monday 21st' from workspace.

(netzob:3361): Gtk-WARNING **: Content added to the action area of a dialog using header bars

(netzob:3361): Gtk-WARNING **: Content added to the action area of a dialog using header bars
Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged.
Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged.
[MainThread]2016-03-21 09:52:37,057 - PCAPImporter - INFO - Starting import from /root/Desktop/captures/3tickets.pcap (linktype:/root/Desktop/captures/3tickets.pcap)
[MainThread]2016-03-21 09:52:37,255 - PCAPImporter - WARNING - Cannot import one of the provided packets since its layer 3 is unsupported (Only IP is currently supported, packet ethernet type = 2054)
Gtk-Message: GtkDialog mapped without a transient parent. This is discouraged.
[MainThread]2016-03-21 09:52:44,969 - Workspace - INFO - Save the config file of the workspace New Workspace in /root/workspace/netzob/workspace.xml
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/netzob/UI/Vocabulary/Controllers/VocabularyController.py", line 261, in symbolListTreeViewSelection_changed_cb
self._view.updateSymbolProperties()
File "/usr/lib/python2.7/dist-packages/netzob/UI/Vocabulary/Views/VocabularyView.py", line 623, in updateSymbolProperties
self.updateSymbolVariableDefinition()
File "/usr/lib/python2.7/dist-packages/netzob/UI/Vocabulary/Views/VocabularyView.py", line 629, in updateSymbolVariableDefinition
variableDisplayerController.run(self.messagesDistributionSymbolViewport)
File "/usr/lib/python2.7/dist-packages/netzob/UI/Vocabulary/Controllers/VariableDisplayerController.py", line 72, in run
self._view.run(panel)
File "/usr/lib/python2.7/dist-packages/netzob/UI/Vocabulary/Views/VariableDisplayerView.py", line 81, in run
xdotWidget.drawDotCode('\n'.join(dotCode))
File "/usr/lib/python2.7/dist-packages/netzob/Simulator/XDotWidget.py", line 64, in drawDotCode
self.set_dotcode(dotCode)
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1475, in set_dotcode
self.set_xdotcode(xdotcode)
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1490, in set_xdotcode
self.graph = parser.parse()
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1169, in parse
DotParser.parse(self)
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 979, in parse
self.parse_graph()
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 988, in parse_graph
self.parse_stmt()
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1018, in parse_stmt
self.parse_subgraph()
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1001, in parse_subgraph
self.parse_stmt()
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1034, in parse_stmt
self.handle_node(id, attrs)
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 1144, in handle_node
shapes.extend(parser.parse())
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 609, in parse
w = s.read_number()
File "/usr/lib/python2.7/dist-packages/netzob/ExternalLibs/xdot.py", line 496, in read_number
return int(self.read_code())
ValueError: invalid literal for int() with base 10: '136.48'

(netzob:3361): Gdk-ERROR **: The program 'netzob' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadAlloc (insufficient resources for operation)'.
(Details: serial 26600 error_code 11 request_code 130 (MIT-SHM) minor_code 5)
(Note to programmers: normally, X errors are reported asynchronously;
that is, you will receive the error a while after causing it.
To debug your program, run it with the GDK_SYNCHRONIZE environment
variable to change this behavior. You can then get a meaningful
backtrace from your debugger if you break on the gdk_x_error() function.)
Trace/breakpoint trap`

Results of Sequence Alignment (only) in Combination with Clustering not Determenistic

Function splitAlinged() works as expected. When using clusterByAlignment() the results often (if not always) vary. Afaik Netzob uses results of Needleman-Wunsch logarithm as a score for UPGMA (correct me if I'm wrong). Does the alignment inside the symbols happening in a separate step or is the score of the first computation being used for splitting too?

The following example does demonstrate it kinda extremly. When I noticed that last week I used a small trace with a proprietary protocol which ended up in two symbols. The second symbol was always the same while the alignment/spliting inside the first symbol was skipped occasionally. Trying the same trace and script on the machine with a different distro of a colleague resulted in whole different overall alignment then mine.

To reproduce the issue:

$ wget "https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=pgsql.cap.gz" -O pgsql.cap.gz
$ gunzip pgsql.cap.gz
# there are some TCP packets in between you want to get rid of
$ tshark -r pgsql.cap -Y pgsql -w pgsql_filtered.pcap 

Then run tests:

from netzob.all import *                                                          

messages = PCAPImporter.readFile('pgsql_filtered.pcap').values()                  


for i in range(0,50):                                                             
    f = open("result_{0!s}.txt".format(i), 'w')                                   
    symbols = Format.clusterByAlignment(messages, minEquivalence=35, internalSlick=True)
    for symbol in symbols:                                                         
        f.write(str(symbol))                                                      
    # This works like expected:                                                   
    #symbol = Symbol(messages=messages)                                           
    #Format.splitAligned(symbol, doInternalSlick=True)                            
    #f.write(str(symbol))                                                         
    f.close()

Afterwards compare the output files

$ du -b result_*
21219   result_0.txt
22003   result_10.txt
22351   result_11.txt
[...]
vimdiff result_{10,11}.txt

[netzob_web] Access-Control-Allow-Origin Header missing

Hi,
the Access-Control-Allow-Origin header for the following URLs is missing

http://localhost:5000/api/p/1/symbols/
http://localhost:5000/api/p/1/captures/.

Workaround: access it via localhost, indead of 127.0.0.1 or ::1
Greetings

Disable debug information in version 1.0rc1

'Anti-Fa Pepe' on Twitter has reported that he cannot disable debug information in version 1.0rc1.
https://twitter.com/Zython86/status/811648149865439232

Parsing of field fails after manual domain change

My intention is to refine a vocabulary model after it has been inferred by clusterByAlignment. Using ASCII datatype as test case, I stumbled across an error in the MessageParser that I could not trace to the source of the problem. Could you please have a look and help me in debugging?

Here the test script that produces the error:

from netzob.Model.Vocabulary.Domain.Parser.MessageParser import InvalidParsingPathException
from netzob.all import *
import pprint

# do pretty printing of variable contents later
pp = pprint.PrettyPrinter(indent=2)

# import two NTP message payloads that get clustered/aligned by clusterByAlignment
# into one symbol with 13 fields at minEquivalence=60
messages = [
    RawMessage(b'\x1a\x02\n\xec\x00\x00\x0ep\x00\x00\x0f\xea\xc6\x1e\\\x02\xc5\x01\xfe@\xd7j\xd7\x0e\xc5\x02\x04\xec\xee\xd3<R\xc5\x02\x04\xeb\xd8\xfd\x8f\x0c\xc5\x02\x04\xeb\xd8\xff2z'),
    RawMessage(b'\x1a\x02\n\xec\x00\x00\x07\xc3\x00\x00/\x80\xc6\x1e\\\x02\xc5\x01\xf9\x95BP\x82\xcf\xc5\x02\x04\xec\xecB\xee\x92\xc5\x02\x04\xeb\xd25.\xb5\xc5\x02\x04\xeb\xd25\xd6{')
    ]
symbols = Format.clusterByAlignment(messages, minEquivalence=60, internalSlick=True)
# this is the only symbol.
sym = symbols[0]
print(sym)

# print field before any manual change
print(sym.fields[1])
dt = sym.fields[1].domain.dataType

mp = MessageParser()

for msg in sym.messages:
    # test parsing of the message before any manual change
    before = mp.parseMessage(msg, sym)
    pp.pprint(before)
    # this works in all cases. Interestingly one of the parsed field values
    # results in a byte, the other in a str:
    # Field
    # -----------
    # b'\x07\xc3'
    # '\x0ep'
    # -----------

# change the domain type of field 1 into ASCII
sym.fields[1].domain = Data(ASCII(nbChars=(dt.size[0]//8, dt.size[1]//8)))

for msg in sym.messages:
    try:
        after = mp.parseMessage(msg, sym)
        # the parsing fails for one message

        for (a,b) in zip(before,after):
            if not a == b:
                print("field parse mismatch: {} | {}".format(a,b))
    except InvalidParsingPathException:
        print("message parsing failed for message {}".format(msg.data))

The stack trace I get if I remove the try around the parseMessage() is:

Traceback (most recent call last):
  File "#11_change-field-domain.py", line 36, in <module>
    print(sym.fields[1])
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Model/Vocabulary/AbstractField.py", line 702, in __str__
    result = self.getCells(encoded=True)
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Common/Utils/Decorators.py", line 188, in wrapped_f
    return func(*args, **kwargs)
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Model/Vocabulary/AbstractField.py", line 281, in getCells
    return DataAlignment.align(data, self, encoded=encoded)
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Common/Utils/Decorators.py", line 188, in wrapped_f
    return func(*args, **kwargs)
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Common/Utils/DataAlignment/DataAlignment.py", line 260, in align
    return dAlignment.execute()
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Common/Utils/DataAlignment/DataAlignment.py", line 163, in execute
    alignedMsg = next(mp.parseRaw(d, targetedFieldLeafFields))
  File "/home/stephan/REUP-common/Tools/netzob-official/netzob/src/netzob/Model/Vocabulary/Domain/Parser/MessageParser.py", line 241, in parseBitarray
    TypeConverter.convert(bitArrayToParse, BitArray, Raw)))

Hash generation for Symbol generates a lot of collisions

The function for hash generation (__hash__()) of Symbol objects uses the name property as input. This leads to seemingly identical objects just if their name is the same, e. g. the default name "Symbol". Moreover this violates Python's assumption that an object never changes its hash within its lifetime (name is set-able).

This makes it impossible to use a Symbol as identifier, e. g. as a key in a dict or member of a set: The last inserted Symbol with a name identical to any of the existing ones in the data structure will be replaced.

Initiates Web-API architecture

Netzob should exposes a Web-API. An architecture based on flask-rest could be used for this.

Symbol is not printed correctly

After I split the messages by using Format.splitDelimiter function, the resulted symbol cannot be printed because of an error.

This is how it is called.

This is the error message.

I read the related code and found out that it can be correctly printed if I remove "decode('utf-8')" in line 186. It seems that it is not always utf-8 string. May I ask why the decode is needed?

Please focus on python3

The python3 branch currently fails to build on x64_arch py3.5.1 and has ony 62% coverage.
Py3 is 8 years old, please move forward from the deprecated and outdated 2.7

netzob interface

Does netzob have an interface?

pictures in doc

images in doc do not exist

Property functions (getter and setter) of memory in class Memory superfluous

The getter and setter functions of the property __memory in class Model.Vocabulary.Domain.Variables.Memory.Memory (see below) are dysfunctional: They expose the internal property __memory in such a way so that the getters and setter become superfluous. In case of the setter it makes it even inefficient, since it copies the dict instead of referencing it.

If copying was intended behaviour, this should be documented. In this case, a more efficient way of doing this would be to replace the complete function body by either:
self.__memory = dict(memory) or self.__memory = memory.copy()

Snippet of the referred functions in Model.Vocabulary.Domain.Variables.Memory.Memory:

    @property
    def memory(self):
        """The content of the memory is stored in this dict().

        :type: :class:`dict`
        """
        return self.__memory

    @memory.setter
    def memory(self, memory):
        self.__memory = dict()
        for k, v in list(memory.items()):
            self.__memory[k] = v

The TypeEncodingFunction doesn't seem to set the endianness correctly

When parsing a message I would like to change the endianness of an integer when it is displayed by netzob, but this does not seem to work. For example:

>>> from netzob.all import *
>>> 
>>> mesg=RawMessage("hello\x00\x00\x00\x01")
>>> f1=Field(ASCII("hello"))
>>> f2=Field(Integer(unitSize=AbstractType.UNITSIZE_32))
>>> s = Symbol(fields=[f1,f2], messages=[mesg])
>>> print s
Field   | Field             
------- | ------------------
'hello' | '\x00\x00\x00\x01'
------- | ------------------
>>> f2.addEncodingFunction(TypeEncodingFunction(Integer, unitSize=AbstractType.UNITSIZE_32, endianness=AbstractType.ENDIAN_LITTLE))
>>> print s
Field   | Field   
------- | --------
'hello' | 16777216
------- | --------
>>> f2=Field(Integer(unitSize=AbstractType.UNITSIZE_32))
>>> f2.addEncodingFunction(TypeEncodingFunction(Integer, unitSize=AbstractType.UNITSIZE_32, endianness=AbstractType.ENDIAN_BIG))
>>> s = Symbol(fields=[f1,f2], messages=[mesg])
>>> print s
Field   | Field   
------- | --------
'hello' | 16777216
------- | --------

I believe I've traced the error to netzob/src/netzob/Common/Models/Vocabulary/Functions/EncodingFunctions/TypeEncodingFunction.py - line 98 which should be changed to
return TypeConverter.convert(data, BitArray, self.type, dst_unitSize=self.unitSize, dst_endianness=self.endianness, dst_sign=self.sign)

After making this change I get the expected results:

>>> from netzob.all import *
>>> mesg=RawMessage("hello\x00\x00\x00\x01")
>>> f1=Field(ASCII("hello"))
>>> f2=Field(Integer(unitSize=AbstractType.UNITSIZE_32))
>>> s = Symbol(fields=[f1,f2], messages=[mesg])
>>> print s
Field   | Field             
------- | ------------------
'hello' | '\x00\x00\x00\x01'
------- | ------------------
>>> f2.addEncodingFunction(TypeEncodingFunction(Integer, unitSize=AbstractType.UNITSIZE_32, endianness=AbstractType.ENDIAN_LITTLE))
>>> print s
Field   | Field   
------- | --------
'hello' | 16777216
------- | --------
>>> f2=Field(Integer(unitSize=AbstractType.UNITSIZE_32))
>>> f2.addEncodingFunction(TypeEncodingFunction(Integer, unitSize=AbstractType.UNITSIZE_32, endianness=AbstractType.ENDIAN_BIG))
>>> s = Symbol(fields=[f1,f2], messages=[mesg])
>>> print s
Field   | Field
------- | -----
'hello' | 1    
------- | -----

This issue may also be related to issue #14

Missing pip dependancies in requirements.txt

Missing numpy, wheel, and flask_restplus in requirements.txt, for the web interface.

Also, version of werkzeug is outdated. Need to upgrade to at least version Werkzeug==0.14.1.

Add a FileImporter

We should provide a simple solution to import the content of a file as a set of messages.

ClusterByAlignment error

From the example in [http://doc.netzob.org/en/latest/developer_guide/API/netzob.Inference.Vocabulary.FormatOperations.html#module-netzob.Inference.Vocabulary.FormatOperations.ClusterByAlignment]

from netzob.all import *

pseudos = ["zoby", "ditrich", "toto", "carlito"]
cities = ["Paris", "Munich", "Barcelone", "Vienne"]
ips = ["192.168.0.10", "10.120.121.212", "78.167.23.10"]
# Creation of the different types of message
msgsType1 = [ RawMessage("hello {0}, what's up in {1} ?".format(pseudo, city)) for pseudo in pseudos for city in cities]
msgsType2 = [ RawMessage("My ip address is {0}".format(TypeConverter.convert(ip, IPv4, Raw))) for ip in ips]
msgsType3 = [ RawMessage("Your IP is {0}, name = {1} and city = {2}".format(TypeConverter.convert(ip, IPv4, Raw), pseudo, city)) for ip in ips for pseudo in pseudos for city in cities]
messages = msgsType1+msgsType2+msgsType3
clustering = ClusterByAlignment()

It reports: NameError: name 'ClusterByAlignment' is not defined

After adding from netzob.Inference.Vocabulary.FormatOperations import ClusterByAlignment
It reports: TypeError: 'module' object is not callable

Warning during dev installation

I installed the prerequisites succesfully.
When I manually do the command python setup.py develop --user, a lot of warnings rise up.
Take a look on this screenshot:

These warnings seems that create no problem for the installation, but maybe must be solved.
Thanks.

Length infered for dynamic length fields sometimes is wrong

The maximum field length infered for dynamic fields by
netzob.Inference.Vocabulary.FormatOperations.FieldSplitAligned.FieldSplitAligned.FieldSplitAligned#_mergeAlign
into the tuple (entryVal, entryDyn) from variable splited in line 252 of
FieldSplitAligned.py
is not correct for "some" inferences like:

<class 'ScapyDissector_ntp.NTPi_Symbol2'>
fieldlengths netzob from domain
[8, 8, 8, 16, 8, 24, 8, 8, 16, 224, 80]
408
fieldlengths netzob counted in value
[8, 8, 8, 16, 8, 24, 0, 8, 16, 224, 64]
384

The values at the bottom are correct, i. e. 64 instead of 80.

I will provide some minimum working example to verify in the near future.

Failing to install on windows?

So I have tried building/installing netzob on my windows box and I keep getting this error and I can't seem to figure out if its me or the code.

I am not very familiar with python so I am sorry in advanced.

This is there error when running python setup.py install
`
warning: no previously-included files matching '.tmp' found under directory ''

warning: no files found matching '*.py' under directory 'src\netzob_plugins'
no previously-included directories found matching 'src\netzob_plugins**\build'

writing manifest file 'Netzob.egg-info\SOURCES.txt'
installing library code to build\bdist.win32\egg
running install_lib
running build_py
Update build\lib.win32-2.7\netzob\NetzobResources.py with BID = "81e2138d-ade9-4
8e8-9d03-ed6fc119cf24"
running build_ext
building 'netzob._libNeedleman' extension
C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo /Ox
/MD /W3 /GS- /DNDEBUG -DBID="81e2138d-ade9-48e8-9d03-ed6fc119cf24" -Ilib\include
s -Ilib\includes\Py_lib -IC:\python27\include -IC:\python27\PC /Tclib\interface
Interface.c /Fobuild\temp.win32-2.7\Release\lib\interface\Interface.obj -std=c99
-Wall -Wextra -Wunused -Wsign-compare -Wstrict-prototypes -Wuninitialized -Wsha
dow -Wpointer-arith -O2
cl : Command line error D8021 : invalid numeric argument '/Wextra'
error: command 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\BIN\c
l.exe' failed with exit status 2`

Any help would be awesome.

[Feature] Floating Point Type

Hi,
a floating point type is missing in

Model/Vocabulary/Types

Greetings

[netzob] AttributeError: 'module' object has no attribute 'ARP'

Hi,

When I want to import a pcap, the program output this errror:

Traceback (most recent call last):
  File "reverse_protocol.py", line 311, in <module>
    main()
  File "reverse_protocol.py", line 192, in main
    importLayer=4).values()
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Common/Utils/Decorators.py", line 188, in wrapped_f
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/PCAPImporter.py", line 515, in readFile
    nbPackets, mergePacketsInFlow)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Common/Utils/Decorators.py", line 188, in wrapped_f
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/PCAPImporter.py", line 485, in readFiles
    return importer.readMessages(filePathList,bpfFilter, importLayer, nbPackets, mergePacketsInFlow)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Common/Utils/Decorators.py", line 188, in wrapped_f
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/PCAPImporter.py", line 442, in readMessages
    self.__readMessagesFromFile(filePath, bpfFilter, nbPackets)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Common/Utils/Decorators.py", line 188, in wrapped_f
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/PCAPImporter.py", line 199, in __readMessagesFromFile
    packetReader.loop(nbPackets, self.__packetHandler)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/PCAPImporter.py", line 248, in __packetHandler
    etherType) = self.__decodeLayer2(header, payload)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/PCAPImporter.py", line 301, in __decodeLayer2
    layer2 = l2Decoder.decode(payload)
  File "/usr/local/lib/python3.4/dist-packages/Netzob-1.0.2-py3.4-linux-x86_64.egg/netzob/Import/PCAPImporter/ImpactDecoder.py", line 68, in decode
    elif e.get_ether_type() == ImpactPacket.ARP.ethertype:
AttributeError: 'module' object has no attribute 'ARP'

My bpfFilter:
not (arp or host 192.168.56.255 or host 216.58.210.174 or host 212.106.221.82 or host 212.106.221.81 or host 212.106.221.76 or host 8.248.185.254 or host 216.58.210.164 or host 216.58.210.163 or host 8.8.8.8)

In few pcaps file, the program works properly.

Thanks in advance!

Multiple file import is not working

When multiple files are selected to be imported, only one of them is retained.

clusterByAlignment fails in computeSimilarityMatrix on RawMessages

Clustering and aligning fails with a SIGSEV for manually provided RawMessages as can be tested by the following example:

from netzob.all import *
m1=RawMessage("123")
m2=RawMessage("345")
messages = [m1, m2]
Format.clusterByAlignment(messages)

I tracked it down to the C-implemented ScoreComputation-library, to be exact the following line 192 in ClusterByAlignment.py:

        (listScores) = _libScoreComputation.computeSimilarityMatrix(
            self.internalSlick, self._cb_executionStatus, self._isFinish,
            debug, wrapper)

At this point I got stuck with debugging C inside of python for now.

To verify, other message contents failed also:

m1=RawMessage("\x41\x42\xab\xac\x09\x70\x95\xcc\xef")
m2=RawMessage("\x41\x42\x0a\x70\x03\x8f\x23\x5f")

or

m1=RawMessage("FGHJERTZ")
m2=RawMessage("CsdJERsd")

Cannot export to peach pit

Hi,

I cannot export to peachpit. Installed 0.4.1 today using the .deb, installed using dpkg. No known un-met dependencies. Installed on Kali, which is Debian-based. When I try to export, nothing happens, but the console I used to start netzob shows the attached error.

The tutorial is still python2, but the tool is now compiled by python3

I just checked the tutorial in netzob/doc/documentation/source/tutorials/discover_features.rst, and find out that the tutorial is still python2. To be run in python3, the parameter of print function should be put in parathesis. And, the xrange is now range.

Meanwhile, the type checking function in "src/netzob/Common/Utils/Decorators.py" around line 181 is too strict. I have to remove the check to run the tutorial.

[netzob_web] 404 for static files

Hi,
i get a 404 for the following requests:

GET http://127.0.0.1:5000/static/libs/datatables.net/js/jquery.dataTables.min.js 
GET http://127.0.0.1:5000/static/libs/datatables.net-select/js/dataTables.select.min.js 
GET http://127.0.0.1:5000/static/libs/jpillora/jquery.rest/dist/1/jquery.rest.min.js 

fixed by adding the js files from : https://datatables.net
and from https://github.com/jpillora/jquery.rest
Greetings

[Feature] Metadata to see request/response

Hi,
if a symbol is printed, metadata, to see if it is a request or a response, would be awseome.
This could look like this:

>>> '\r\x02\x00\x00\x00\x10' | '\x14\x1a\x14\n\x12State:State.SENSOR'
<<< '\r\x02\x00\x00\x00\x10' | b'\x0b\x1a\x12\n\x0btemperature\x15\xe8\x94\xd8A'

Greetings

TypeEncodingFunction is not effective with Decimals

It should be possible to display the data of a field encoded in Decimal.
Besides, this encoding should follow the specified unitSize.

AbstractType.unitSize value is a string instead of an int

The available values of AbstractType.unitSize are :
UNITSIZE_1 = '1'
UNITSIZE_4 = '4'
UNITSIZE_8 = '8'
UNITSIZE_16 = '16'
UNITSIZE_32 = '32'
UNITSIZE_64 = '64'

Why aren't they int values?
UNITSIZE_1 = 1
UNITSIZE_4 = 4
UNITSIZE_8 = 8
UNITSIZE_16 = 16
UNITSIZE_32 = 32
UNITSIZE_64 = 64

Comparing int values is faster than str.
Furthermore, in splitStatic() of Inference/Vocabulary/Format.py the docstring tells that unitSize parameter is an int.
And in RelationFinder._generateDataValues() we can see the conversion from int to str then from str to int to get the good unitSize...

Add a method to split a field in two

It should be easier to split a field into two sub fields.
Split could be based on an index or on a value.

Implements a WebAPISessionController

This new controller should be triggered with the web-interface startup. It controls the web-api used by the web interface.

Sequence alignment fails in many cases

In many cases the alignment fails. The following exception is thrown when getCells() is called on a symbol after alignment:
[...]File "/usr/lib/python2.7/site-packages/Netzob-1.0_git-py2.7-linux-x86_64.egg/netzob/Common/Utils/DataAlignment/ParallelDataAlignment.py", line 187, in execute raise Exception("At least one data ({0}) has not been successfully computed by the alignment".format(repr(d))) Exception: At least one data ('\x01\x01\x06\x00j\x98/\x03\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00[...]

To reproduce the issue you can use the following network trace:
https://violentinter.net/static/tmp/net-2009-11-13-09:24_dhcp2_trimmed.pcap
It's an publicly available trace originally from http://digitalcorpora.org/corp/nps/scenarios/2009-m57-patents/net/net-2009-11-13-09:24.pcap.gz
The first linked pcap was filtered for dhcp traffic and then trimmed to 100 packets.
To trigger the exception the following code can be used:

from netzob.all import *
messages = PCAPImporter.readFile('net-2009-11-13-09:24_dhcp2_trimmed.pcap').values()
symbol = Symbol(messages=messages)
Format.splitAligned(symbol, doInternalSlick=True)
print symbol.getCells() # here the explosion happens

The exception of course is triggered using clusterByAlignment() too since splitAlignment() is called after clustering.
I can reproduce this issue with a lot of different public traces. A few of them work well, most fail.

Possible to influence:
I noticed that that using "doInternalSlick=True" prevent the issue occurring for some traces. Also varying the equivalence sometimes help.

Thanks in advance

Remove Regex from Netzob

Old netzob uses to rely on Regex to align and parse messages.
This was improved by the new parser which do not requires anymore Regex to perform.

Thus, dependencies (such as re) and classes related to regexes (such as NetzobRegex) should be removed.

bintrees is deprecated, suggests sortedcontainers

Hi, I'm the author of sortedcontainers and noticed that you're using the bintrees module. If you look at the PyPI page for bintrees then you'll see that development has stopped and sortedcontainers is recommended instead.

I'd love to see sortedcontainers used in more places. If I created a pull request which replaced bintrees with sortedcontainers, would that be merged?

install error

Hi , i want to install netzob in ubuntu16.04-64bit ,python3 verion is 3.5.2
i installed prerequisites as readme.ret asked.
then i put "python3 setup.py develop --user",it appeared:
ImportError: No module named 'numpy'
i used command "pip3 install numpy‘’ and reinstalled "python3 setup.py develop --user".it showed:

`Processing pcapy-0.10.10.tgz

Writing /tmp/easy_install-7jzb1zyi/pcapy/setup.cfg

Running pcapy/setup.py -q bdist_egg --dist-dir /tmp/easy_install-7jzb1zyi/pcapy/egg-dist-tmp-xb0m1kom

cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++

pcapdumper.cc:11:18: fatal error: pcap.h: No such file or directory

compilation terminated.

error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
`
i found pcapy is in requirements.txt and will be installed automatically ,what's wrong with it?
thanks!

README misses required dependency: libpcap0.8-dev

On executing setup.py compilation of pcap importer parts fails with missing pcap.h-file.

Debian, e. g., provides the required header file in package libpcap0.8-dev.

from werkzeug.contrib.cache import (nullcache, simplecache, memcachedcache, importerror: cannot import name 'uwsgicache'

from werkzeug.contrib.cache import (nullcache, simplecache, memcachedcache, import error: cannot import name 'uwsgicache'
The werkzeug version is not suitable for netzob_web. To address this problem, you need to upgrade the werkzeug, use$pip3 install werkzeug --upgrade.

Have an issue when installing in osx

I'm just trying to install netzob in Sierra 10.12.6 by

$ pip3 install netzob

And get this:

Collecting arpreq==0.3.1 (from netzob)
Using cached arpreq-0.3.1.tar.gz
Installing collected packages: arpreq, netzob
Running setup.py install for arpreq ... error
Complete output from command /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;file='/private/var/folders/fs/4gkfdnfd49gggp_f0mr7lyxc0000gn/T/pip-build-5gf_ixyt/arpreq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/fs/4gkfdnfd49gggp_f0mr7lyxc0000gn/T/pip-tlt_1uj2-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_ext
building 'arpreq' extension
creating build
creating build/temp.macosx-10.6-intel-3.6
creating build/temp.macosx-10.6-intel-3.6/src
/usr/bin/clang -fno-strict-aliasing -Wsign-compare -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c src/arpreq.c -o build/temp.macosx-10.6-intel-3.6/src/arpreq.o -std=c99
src/arpreq.c:240:39: error: use of undeclared identifier 'SIOCGIFHWADDR'
if (ioctl(st->socket, SIOCGIFHWADDR, &ifreq) == -1) {
^
src/arpreq.c:243:49: error: no member named 'ifr_hwaddr' in 'struct ifreq'
memcpy(&mac_address, &ifreq.ifr_hwaddr, sizeof(mac_address));
~~~~~ ^
/usr/include/secure/_string.h:65:33: note: expanded from macro 'memcpy'
__builtin___memcpy_chk (dest, src, len, __darwin_obsz0 (dest))
^~~
src/arpreq.c:251:28: error: no member named 'arp_dev' in 'struct arpreq'
strncpy(arpreq.arp_dev, ifa->ifa_name, IFNAMSIZ);
~~~~~~ ^
/usr/include/secure/_string.h:119:28: note: expanded from macro 'strncpy'
__builtin___strncpy_chk (dest, src, len, __darwin_obsz (dest))
^~~~
src/arpreq.c:251:28: error: no member named 'arp_dev' in 'struct arpreq'
strncpy(arpreq.arp_dev, ifa->ifa_name, IFNAMSIZ);
~~~~~~ ^
/usr/include/secure/_string.h:119:59: note: expanded from macro 'strncpy'
__builtin___strncpy_chk (dest, src, len, __darwin_obsz (dest))
^~~~
/usr/include/secure/_common.h:39:54: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
^~~~~~
src/arpreq.c:252:35: error: use of undeclared identifier 'SIOCGARP'
if (ioctl(st->socket, SIOCGARP, &arpreq) == -1) {
^
5 errors generated.
error: command '/usr/bin/clang' failed with exit status 1

----------------------------------------

Command "/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 -u -c "import setuptools, tokenize;file='/private/var/folders/fs/4gkfdnfd49gggp_f0mr7lyxc0000gn/T/pip-build-5gf_ixyt/arpreq/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /var/folders/fs/4gkfdnfd49gggp_f0mr7lyxc0000gn/T/pip-tlt_1uj2-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/fs/4gkfdnfd49gggp_f0mr7lyxc0000gn/T/pip-build-5gf_ixyt/arpreq/

CRC detection

I am reversing a Scada protocol using Netzob. I have already reversed it manually and am trying to automate some tasks. One of the fields I have found using the Format.SplitStatic() method is the CRC of several other fields. I want Netzob to "understand" this relation. Hence I used the CorrelationFinder.find() method on a symbol. It returns plenty of relations, however these only have SizeRelation as a relation type. Nothing indicating a CRC. Am I proceeding the right way? Is there a way to manually tell Netzob "Hey this field is the CRC of these fields" so that it will take it in account when using the symbol.specialize() method?

Thanks in advance

ImpactPacket.ARP and ARPDecoder are missing

Lines 68 to 70 in ImpactDecoder call ImpactPacket.ARP and ARPDecoder in next and master branches, but both Classes are missing in netzob/Import/PCAPImporter.

Invert the meaning of encode() and decode() in types classes

For me, the meaning of encode() and decode() appears to be inverted.

For example:
raw = Integer.decode(23)
should be written:
raw = Integer.encode(23)
as the objective is to encode an integer into a raw stream.

The documentation builder (sphinx) shows import errors

When triggering the documentation build with the command::

python setup.py build_sphinx

some import errors are reported such as

netzob/doc/documentation/source/developer_guide/API/netzob.Inference.Grammar.rst:53: WARNING: autodoc: failed to import module u'netzob.Inference.Grammar.LearningAlgorithm'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/sphinx/ext/autodoc.py", line 335, in import_object
    __import__(self.modname)
  File "src/netzob/Inference/Grammar/LearningAlgorithm.py", line 44, in <module>
    from netzob.Inference.Grammar.Oracles.NetworkOracle import NetworkOracle
  File "src/netzob/Inference/Grammar/Oracles/NetworkOracle.py", line 44, in <module>
    from netzob.Common.MMSTD.Dictionary.AbstractionLayer import AbstractionLayer
ImportError: No module named MMSTD.Dictionary.AbstractionLayer

I believe these errors are due to old netzob code hidding in some dark places of the repository.