cisco / joy Goto Github PK

I would like to be able to extract select fields from within an array.

For example, the following dns response.
{ "dns": [{"rn": "pool.ntp.org", "rr": [{"a": "192.168.10.17", "ttl": 35}, {"a": "192.168.12.251", "ttl": 35}, {"a": "192.168.12.251", "ttl": 35}

--select "dns[].rr[].a"
would extract all of the "a" records from the dns response.

Hi,

I am facing an unknown issue with sleuth.

First, I use joy to generate the output file with "joy bidir=1 dist=1 classify=1 ../benign/*.pcap > ./benign_classify.gz", which contains 3929 lines.

Second, I do "sleuth benign_classify.gz --select "p_malware" --where "p_malware > 0.49" > temp.txt", then temp.txt has 57 lines. And if do "sleuth benign_classify.gz --select "p_malware" --where "p_malware < 0.5" > temp.txt", it has 514 lines in the temp.txt. Hence now you can see 57+514 is way less than 3929, does this mean sleuth has bug?

By the way, I have written a very simple python program to verify it as the following, and the results match 3929 lines.

Best wishes,
Xiaoban

import json
import sys
import gzip
if __name__ == "__main__":
	if len(sys.argv) != 4:
		print "Error using this program"
		exit()
	json_file = sys.argv[1]
	mode = sys.argv[2]
	target = float(sys.argv[3])
	count = 0
	with gzip.open(json_file,'r') as fp:  
		for line in fp:
			tmp = json.loads(line)
			if 'version' in tmp:
				continue
			t = float(tmp['p_malware'])
			if mode == ">":
				if t > target:
					count = count + 1
			elif mode == ">=":
				if t >= target:
					count = count + 1
			elif mode == "<":
				if t < target:
					count = count + 1
			elif mode == "<=":
				if t <= target:
					count = count + 1
		print count

Is there a way we can make Joy analyze the CAPWAP traffic payload (assuming its unencrypted) ?

Quick question - does anybody know how to make use of VPP in Joy? Is there any doc on this?

I just saw a line in the release notes in release-3.0.0 highlight the VPP integration but no any more info found:

Implemented a Vector Packet Processing integration scheme to utilize VPP native infrastructure when building that integration.

The interface_list_get() method doesn't set the active flag and the tool displays interfaces as down despite it being active. The below patch should fix the issue.

diff --git a/src/joy.c b/src/joy.c
index a4fa5bd..488f05e 100644
--- a/src/joy.c
+++ b/src/joy.c
``
@@ -233,6 +233,7 @@ static unsigned int interface_list_get (struct interface ifl[IFL_MAX]) {
if (ioctl(sock, SIOCGIFHWADDR, &ifr) == 0) {
memcpy(ifl[i].mac_addr, ifr.ifr_hwaddr.sa_data, 6);
strncpy(ifl[i].name, it->ifr_name, IFNAMSIZ);

•                   ifl[i].active = 1;
                       i++;
                       break;
                   }
  

``

Hi,
I'm working using Joy to analyse udp streams (Specially traffic like NTP).
I found that, joy is merging multiple UDP packets into few record entries.

In my case I had 6 packets as shown follows

But I got only 3 records like this.

I would like to know on what criteria, joy is merging the UDP packets and is there anyway to write them as individual entries?

Hello，I saw these codes and comments while reading tls.c

/*
 * The maxiumum allowed length of a serial number is 20 octets
 * according to RFC5290 section 4.1.2.2. We give some leeway
 * for any non-conforming certificates.
 */
#define MAX_CERT_SERIAL_LENGTH 24
#define MAX_HANDSHAKE_LENGTH 11000

The comments in the section mentioned RFC 5290, but I reviewed RFC 5290 and found no relevant content.I checked the relevant information, which appeared in RFC 5280 section 4.1.2.2.

Given the uniqueness requirements above, serial numbers can be
expected to contain long integers. Certificate users MUST be able to
handle serialNumber values up to 20 octets. Conforming CAs MUST NOT
use serialNumber values longer than 20 octets.

Please check if the comments in this place are correct. Thank you.

Is it possible to report per packet information (ppi) for more that 200 packets for a flow? The MAX_NUM_PKT_LEN is defined to be 200.

If the flow has more than 200 packets, does Joy save packet information for all the packets or does it just save the packet information for the first 200 packets?

Thanks a lot,

Xiaoming

Hello,

I am using Joy to inspect various .pcap files that were generated by collecting network data on a Hadoop cluster. tcpdump was used and only packets on the local network - on the cluster - were collected.
When Joy is used on such .pcap files, instead of seeing various flows among the local network nodes ONLY, I see various other strange addresses, and no local flows as I'd expect.

Here I generate the .gz archive using joy from a .pcap file.

Here you can see the contents of the .pcap in Wireshark

And yet, the output has addresses which are not seen in Wireshark at all.

Why does this occur?

Hello,
I see in model.py that there is some code for creating logistic regression paramteres with TLS information included, but there is no option to enable that. Is there some reason for this? Does the code otherwise work?

when i looking for a "tls switch" ,I find the in file "tls.h" there is a macro definition named "tls_filter",but this macro definition is not being called anywhere.
also in file "config.c", there is no tls option.

i am wondering, how this proc worked when tls switch is opened.

thanks, for your attention.

Is it possible to have a filter that excludes IP address or ports?
e.g. --where "ip != 1.1.1.1"

I would like to know how byte distribution is calculated. Is it any statistical measure applied like mean, standard deviation. Whenever we give a Pcap file and if we select

./query.py capture.gz --select "bd"

it will give 256 values. I would like to know how it is calculated?

Issue Description

Joy tool was used to process .pcap files obtained from publicly available data set[1]. The process went smoothly for majority of the .pcap files. One of the .gz file generated via joy processing threw an unexpected error. After analyzing the crux of this problem, it is noted that this error caused due to a malformed JSON.

Steps to re-produce the issue

Below are the steps to reproduce this issue for further analysis.

1. Download the malware capture 2013-08-20_capture-win6.pcap file from Stratosphereips malware repositiry dataset[2][3]
2. Use Joy to process the .pcap file by executing following command in your Joy instance
   ./joy bidir=1 http=1 tls=1 dns=1 ppi=1 output=output.gz 2013-08-20_capture-win6.pcap
3. Read each line from output.gz file via the function LoadJoy().

def LoadJoy(filename):
    """Loads a .gzip file extracted from PCAP using CISCO's Joy tool
    Arguments:
    filename -- gzip file to open
    """
    with gzip.open(filename) as infile:
        iterFile = iter(infile)
        headerData = json.loads(next(iterFile).decode('utf-8').replace('\\',''))
        for bline in iterFile:
            try:
                line = bline.decode('utf-8')
                dj = json.loads(line.replace('\\',''))
            except:
                print("Error {} and dj is {}".format(line, dj))
    return 1

file = 'output.gz'
LoadJoy(file)

Note: This function raise an exception if there's a JSON parse error.

4. This function caught multiple JSON parse errors.

5. After processing one of the result using an online JSON beautifier tool[4], it is noted that all malformed JSONs gives the same error. And the error is :

Parse error on line 1:
...netAuthority.crt."},],"validity_not_befo
-----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '[', got ']'

(There could be be several other types of JSON read errors. This is the error that I encountered after processing 2013-08-20_capture-win6.pcap file)

References

[1]https://www.stratosphereips.org/datasets-overview/
[2]https://www.stratosphereips.org/datasets-malware/
[3]https://mcfp.felk.cvut.cz/publicDatasets/CTU-Malware-Capture-Botnet-6/
[4]https://codebeautify.org/jsonviewer

When compiling the latest version of joy, I am seeting similar errors as #121 . My OS is Ubuntu 16.04 and my OpenSSL version is 1.0.2l. I have ran the config script before the building and the necessary packages are in place. Here is the error output:

Building joy ...
gcc -Werror -Wall -Wbad-function-cast -Wchar-subscripts -Wcomment -Wdeprecated-declarations -Wdisabled-optimization -Wdiv-by-zero -Wendif-labels -Wformat -Wformat-extra-args -Wformat-security -Wformat-y2k -Wimplicit -Wimplicit-function-declaration -Wimplicit-int -Wimport -Winline -Winvalid-pch -Wmain -Wmissing-braces -Wmissing-format-attribute -Wmissing-noreturn -Wmultichar -Wnested-externs -Wnonnull -Wparentheses -Wpointer-arith -Wreturn-type -Wsequence-point -Wshadow -Wsign-compare -Wstrict-aliasing -Wstrict-prototypes -Wswitch -Wswitch-default -Wswitch-enum -Wsystem-headers -Wtrigraphs -Wunknown-pragmas -Wunused -Wunused-function -Wunused-label -Wunused-parameter -Wunused-value -Wunused-variable -Wwrite-strings -Wno-pointer-sign -Wextra -Wno-nullability-completeness -Wno-cast-align -Wno-expansion-to-defined -fPIC -fPIE -D_FORTIFY_SOURCE=2 -O -g -DLINUX=1 -DVERSION="4.0.1" -DUSE_GZIP -DCOMPRESSED_OUTPUT=1 -o "/home/techie/joy/bin/joy" -I ./include -I "/usr/local/lib/python2.7/dist-packages/external/boringssl/src/include" -I "/usr/local/lib/python2.7/dist-packages/external/curl/include" joy.c p2f.c config.c osdetect.c anon.c pkt_proc.c nfv9.c tls.c classify.c radix_trie.c hdr_dsc.c procwatch.c addr_attr.c addr.c wht.c http.c str_match.c acsm.c dns.c example.c updater.c ipfix.c ssh.c ike.c salt.c parson.c fingerprint.c ppi.c utils.c dhcp.c payload.c proto_identify.c -L "/usr/lib/x86_64-linux-gnu" -lpcap -lm -lcrypto -lcurl -lpthread -lz
/tmp/ccV6YBxM.o: In function tls_x509_get_validity_period': /home/techie/joy/src/tls.c:487: undefined reference to BIO_get_mem_ptr'
/home/techie/joy/src/tls.c:503: undefined reference to BIO_reset' /home/techie/joy/src/tls.c:518: undefined reference to BIO_get_mem_ptr'
/tmp/ccV6YBxM.o: In function tls_x509_get_extensions': /home/techie/joy/src/tls.c:1045: undefined reference to BIO_get_mem_ptr'
collect2: error: ld returned 1 exit status
Makefile:132: recipe for target 'joy' failed
make[1]: *** [joy] Error 1
make[1]: Leaving directory '/home/techie/joy/src'
Makefile:58: recipe for target 'all' failed
make: *** [all] Error 2

Seeing the error, I tried to directly build the deb package as suggested by @brilong. Running "./build_pkg -t deb -v 1.0", it kept asking me to install the missing packages as follows.

-t was triggered with option deb
-v was triggered with option 1.0
Please install the missing packages. All required packages
can be installed as follows:
apt-get install gcc git libcurl3 libcurl4-openssl-dev libpcap0.8 libpcap-dev libssl1.0.0 libssl-dev make python python-pip ruby ruby-ffi zlib1g zlib1g-dev

However, I have installed all the packages as directed.
Could you provide any suggestions for resolving this compiling issue?

Hello！I am having problems parsing TLS data，I don't know where the problem is.
Use version：4.0.0
Operating system：CentOS Linux release 7.5.1804 (Core)、Ubuntu16.04
Configuration file：

output=output/gz
bidir=1
dist=1
classify=1
tls=1
entropy=1
verbosity=1
logfile=output/log/20190129_01.log

I removed the pcap_setfilter() part of the function process_pcap_file() to avoid no VLAN support for data link type.

Use command：

bin/joy -x output/option_config.txt ../DATA/PCAP/

Some data parsing errors were found when viewing the parsing results：

"bytes_out":0,
"packets":[],
"byte_dist":[0,0,0.....],
or
"tls":{"error":"no role"},

However, when I execute a command to process this PCAP file, TLS parsing is normal.
Command at this time：

bin/joy -x output/option_config.txt ../DATA/PCAP/9956.pcap > output/gz/9956.gz

I have reproduced this question many times. The HTTPS data size in the PCAP folder is 6G. I don't know why this problem occurs.Is there any solution?

Looking forward to reply, thank you.

OS Version: macOS High Sierra 10.13

$ ./config
Checking for required dependencies
Checking for libpcap...exists
Checking for libcrypto...exists
Checking for libz...exists
Checking for libcurl...exists
Checking for libpthread...exists
Searching for OpenSSL headers...found
Searching for Curl headers...found
All dependencies found. Issue 'make clean;make' to build the code

$ make
Makefile variables:
buildroot=joy
sysname=DARWIN
version=1.73
cwd=src
gcc -Wall -Wno-deprecated-declarations -g -O3 -DDARWIN=1 -DVERSION="1.73" -o "/Users/anish/Documents/SJSU/CS297/Code/joy/bin/joy" -I ./include -I "/usr/local/Cellar/openssl/1.0.2l/include" -I "/usr/include" joy.c p2f.c config.c osdetect.c anon.c pkt_proc.c nfv9.c tls.c classify.c radix_trie.c hdr_dsc.c procwatch.c addr_attr.c addr.c wht.c http.c str_match.c acsm.c dns.c example.c updater.c ipfix.c ssh.c ike.c ip_id.c salt.c parson.c fingerprint.c ppi.c utils.c dhcp.c -L "/usr/lib" -lpcap -lm -lcrypto -lcurl -lpthread -lz
Undefined symbols for architecture x86_64:
"_X509_get0_signature", referenced from:
_tls_x509_get_signature in tls-7260b4.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [joy] Error 1
make: *** [all] Error 2

I am getting a segmentation fault in nfv9.c when processing a flow record with TLS data types in it. The net flow with et-analytics enabled was generated by a Cisco CSR1000v running on the AWS.

It takes a segmentation fault in nfv9_process_flow_record for any of the TLS types.

ip-10-0-0-106#show version
Cisco IOS XE Software, Version 16.09.01
Cisco IOS Software [Fuji], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.9.1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Tue 17-Jul-18 16:57 by mcpre

See packets 20 & 22 in the attached file.
crs1000.pcap.zip

Matt

Hello,
Joy doesn't parse TLS records that span over multiple packets, e.g handshake record with server hello and multiple certificates. The parsing doesn't pass the condition at tls.c:1787.
Are there any plans to implement this in the near future?

I want to use joy-master (joy 3.0) to extract the flows in the pcap, but I find that it cannot extract the information of incomplete tls flows. For example, the captured flow only includes C2S Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message packet, and S2C new session ticket, change cipher spec, Change Cipher Spec, Hello Request packet, Then the encrypted communication will be established. There is no Client Hello and Server Hello. For this flow, I cannot extract the tls metadata in Change Cipher Spec with joy-master now. But I test the joy-1.7, it is ok to extract the incomplete flows. I use the same command "./joy tls=1 bidir=1 ../../2.pcap |gunzip" to run it.

Can you help me to solve it?

I want to extract the payload data from a pcap file. How to achieve this?

I have gotten the logging to a file, unbuffered working etc... however I am running into an issue where there seems to be a record on start, written to the output location, (instead of the logfile location) that is causing me some issue as the schema is very different from the output records... i.e. This is written:

{"version":"1.72","interface":"intf11","promisc":1,"output": ...

and then next record is

{"sa":"172.17.0.2","da":"192.168.0.105","pr":6,"sp":59518,"

To me, that first example should be in logfile, and the second in output, and out should be somewhat consistent in what is produced... does that make sense?

I do have verbosity = 1 right now, but that shouldn't matter, right? the first record is not packet output, it's summary about the start of the application, or am I looking at this incorrect?

Thanks!

Hello.

I found a heap-buffer-overflow bug in joy.

Please confirm.

Thanks.

OS: CentOS 7 64bit
Version: commit 0931ccf
PoC Download: process_udp.zip

Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./joy $FILE

=================================================================
==3715==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x606000000062 at pc 0x000000539c1e bp 0x7ffe730b5ed0 sp 0x7ffe730b5ec8
READ of size 2 at 0x606000000062 thread T0
    #0 0x539c1d in process_udp /home/karas/joy/src/pkt_proc.c:727:15
    #1 0x539c1d in process_packet /home/karas/joy/src/pkt_proc.c:990
    #2 0x7fbed9a4f55c  (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f55c)
    #3 0x514977 in process_pcap_file /home/karas/joy/src/joy.c:1089:16
    #4 0x512088 in main /home/karas/joy/src/joy.c:1022:28
    #5 0x7fbed826b3f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #6 0x41b8a9 in _start (/home/karas/joy/bin/joy+0x41b8a9)

0x606000000062 is located 2 bytes to the right of 64-byte region [0x606000000020,0x606000000060)
allocated by thread T0 here:
    #0 0x4d9a28 in __interceptor_malloc (/home/karas/joy/bin/joy+0x4d9a28)
    #1 0x7fbed9a5001a  (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x2001a)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/karas/joy/src/pkt_proc.c:727:15 in process_udp
Shadow bytes around the buggy address:
  0x0c0c7fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0c7fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0c7fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0c7fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0c7fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c0c7fff8000: fa fa fa fa 00 00 00 00 00 00 00 00[fa]fa fa fa
  0x0c0c7fff8010: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x0c0c7fff8020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0c7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3715==ABORTING

==========================================
[Acknowledgement]
This work was supported by ICT R&D program of MSIP/IITP. [R7518-16-1001, Innovation hub for high Performance Computing]

Sometimes, instead of the actual first bytes of the body, all that can be seen in the body field is '2e00'. Is this intended?

Hello,

this might not be an issue actually, but in the json files I see the srlt value, which I can't correlate as an acronym or terminology to the documentation. Is this simply a typo for SPLT?

thanks

Thanks for your attention.
I am troubled verymuch. what is file “map.h” doing?

for example when he define EVAL(...), he writed like this :
#define EVAL0(...) VA_ARGS
#define EVAL1(...) EVAL0 (EVAL0 (EVAL0 (VA_ARGS)))
#define EVAL2(...) EVAL1 (EVAL1 (EVAL1 (VA_ARGS)))
#define EVAL3(...) EVAL2 (EVAL2 (EVAL2 (VA_ARGS)))
#define EVAL4(...) EVAL3 (EVAL3 (EVAL3 (VA_ARGS)))
#define EVAL(...) EVAL4 (EVAL4 (EVAL4 (VA_ARGS)))
why? what is he doing?
In file "feature.h" he define so many macors. Only arguments "wht\example\dns\tls..." is resolved in macors "config_all_features_bool(feature_list);" others are resolved in function "config_parse_command"
why he do this? how "config_all_features_bool" worked? what is this argument "feature_list"?

thanks for your attention again.

...tls.c:1023: undefined reference to `BIO_get_mem_ptr'
collect2: error: ld returned 1 exit status
Makefile:93: recipe for target 'joy' failed
make[1]: *** [joy] Error 1
....

Hi,
are there any plans to support IPv6?

Hello.

I found a heap-buffer-overflow bug in joy.

Please confirm.

Thanks.

OS: CentOS 7 64bit
Version: commit 0931ccf
PoC Download: process_tcp.zip

Steps to reproduce:
1.Download the .POC files.
2.Compile the source code with ASan.
3.Execute the following command
: ./joy $FILE

=================================================================
==3713==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x607000000069 at pc 0x00000053bb3e bp 0x7fff13592c70 sp 0x7fff13592c68
READ of size 1 at 0x607000000069 thread T0
    #0 0x53bb3d in process_tcp /home/karas/joy/src/pkt_proc.c:653:58
    #1 0x53bb3d in process_packet /home/karas/joy/src/pkt_proc.c:984
    #2 0x7f6c0504755c  (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x1f55c)
    #3 0x514977 in process_pcap_file /home/karas/joy/src/joy.c:1089:16
    #4 0x512088 in main /home/karas/joy/src/joy.c:1022:28
    #5 0x7f6c038633f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #6 0x41b8a9 in _start (/home/karas/joy/bin/joy+0x41b8a9)

0x607000000069 is located 0 bytes to the right of 73-byte region [0x607000000020,0x607000000069)
allocated by thread T0 here:
    #0 0x4d9a28 in __interceptor_malloc (/home/karas/joy/bin/joy+0x4d9a28)
    #1 0x7f6c0504801a  (/usr/lib/x86_64-linux-gnu/libpcap.so.0.8+0x2001a)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/karas/joy/src/pkt_proc.c:653:58 in process_tcp
Shadow bytes around the buggy address:
  0x0c0e7fff7fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0e7fff7fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0e7fff7fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0e7fff7fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c0e7fff7ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c0e7fff8000: fa fa fa fa 00 00 00 00 00 00 00 00 00[01]fa fa
  0x0c0e7fff8010: fa fa 00 00 00 00 00 00 00 00 00 00 fa fa fa fa
  0x0c0e7fff8020: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
  0x0c0e7fff8030: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff8040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff8050: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3713==ABORTING

==========================================
[Acknowledgement]
This work was supported by ICT R&D program of MSIP/IITP. [R7518-16-1001, Innovation hub for high Performance Computing]

Hi,
I started to explore your DB a bit, and I found out that the content in the json of extension psk_key_exchange_modes (extension 0x002d) is always empty, though it seems like from here:

that you want to extract the value of that extension.
Am i missing something?

Thanks!

current usage message leaves out "http=1" which is required for ohttp and ihttp headers

I tried to use joy to parse a pcap file of raw packets. Unfortunately, it output "error: could not parse filter ip or vlan: no VLAN support for data link type 12". How can I fix that?
If not possible, is there any other tool which can parse pcap of raw packets?

Hello, it would be useful to have a filter for comparing string values, e.g.
--where "tls_sec{seclevel{classification}} = legacy"
or
--where 'tls_sec{seclevel{classification}} = "legacy"'
These don't work ATM

I am curious as to how traffic direction is calculated.

I have .pcap from a lab where there are private IP addresses used.
I used ./bin/pcap2flow bidir=1 http=1 dns=1 tls=1 dist=1 to generate the flows.

Below seems to indicate there are 2664 flows inbound from an external source port of 443 into my lab.

If i look at wireshark, it indicates traffic is outbound.

$ ./joy/query.py --select "sa, sp, da, dp, ib,ip,ob,op,dns" --where 'da=192.168.1.106' ./joy/fullDay.json.gz | wc -l
2664
$ ./joy/query.py --select "sa, sp, da, dp, ib,ip,ob,op,dns" --where 'sa=192.168.1.106' ./joy/fullDay.json.gz | wc -l
6
$ ./joy/query.py --select "sa, sp, da, dp, ib,ip,ob,op,dns" --where 'da=192.168.1.106' ./joy/fullDay.json.gz | head
{
"name": [
{ "sa": "52.x.x.x" , "sp": 443 , "da": "192.168.1.106" , "dp": 46330 , "ib": 2828 , "ip": 31 , "ob": 74 , "op": 31 } ,
{ "sa": "52.x.x.x" , "sp": 443 , "da": "192.168.1.106" , "dp": 46330 , "ib": 2918 , "ip": 32 , "ob": 74 , "op": 32 } ,

Hi,

I am trying to run model.py to generate a new set of params, however I am encountering the following issue. I have double checked that there is indeed malware.gz under malware_train and benign.gz under benign_train, which are genrated by Joy. I see that the numbers of positive and negative are both zero, is there something wrong with opening the generated files?

Thanks very much for your help.
Xiaoban

/joy/analysis$ python model.py -m -l -t -p ../benign_train/ -n ../malware_train/ -o params.txt
Num Positive: 0
Num Negative: 0

Features Used:
Metadata (7)
Packet Lengths (100)
Packet Times (100)
Total Features: 207

/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
Traceback (most recent call last):
File "model.py", line 150, in
main()
File "model.py", line 146, in main
learn_param(data, labels, args.output)
File "model.py", line 49, in learn_param
logreg.train(data, labels)
File "/home/acanets/joy/analysis/classifier.py", line 58, in train
self.logreg.fit(data,labels)
File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/logistic.py", line 1285, in fit
accept_large_sparse=solver != 'liblinear')
File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 756, in check_X_y
estimator=estimator)
File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 552, in check_array
"if it contains a single sample.".format(array))
ValueError: Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.

I would like to silence the warning about missing attributes.

Can this either goto stderr or be silenced with a --quiet option?

After passing a pcap file through joy, I am returned with this error:

error: could not parse filter ip or vlan: no VLAN support for data link type 113

Any help would be appreciated. I can submit further information if needed.

line 1668-1684 in p2f.c is not needed, since line 1628 has do the same thing.

some segment fault show when i set "dns=1",some one can help me ?
in wireshark when it parse a dns protocal may cause problem like "name contains a pointer that loops". when this condition appeared, in this project since it use a recursion to parse dns answer, it cause a segment fault. suggest to solve this bug.

Is there any way to process the packets with 802.1Q Virtual LAN headers using JOY?

Hello,
when joy parses TLS client key exchange message, it includes the length information about the key in the key itself. So if I have a 256 byte RSA premaster secret in my pcap, the clientKeyExchange field will start with 0x0100 and the tls_client_key_length will have the value 2064 instead of 2048. I'm not sure if this is desirable behavior.

$ ./query.py out.json.gz --where "dp=53 & sa=192.168.1.243 & sp=58126" --select "sa, sp, dns"
{
"name": [
{ "sa": "192.168.1.243" , "sp": 58126 , "dns": [{u'rn': u'4-edge-chat.facebook.com', u'rr': [{u'cname': u'star.c10r.facebook.com', u'ttl': 3548}, {u'a': u'179.60.193.3', u'ttl': 39}], u'rc': 8}] }
]
}

The "dns" value is not valid json.

Hi all, I am currently building on a macOS Mojave (10.14.2) machine but I seem to be having some issue at building the package. When I executed

make clean;make

I encountered the following error. My OpenSSL version is 1.0.2q.

../src/fp.c:98:33: error: unused parameter 'header' [-Werror,-Wunused-parameter]
const struct pcap_pkthdr *header,
^
1 error generated.
make[2]: *** [../src/libjoy_la-fp.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

Before that, I executed:

./configure --enable-gzip --with-ssl-dir=/usr/local/ssl/

but this did not seem to generate any error and the Makerfile is created.

Any help would be much appreciated. Thank you.

IS there a way I can have Joy write to a named pipe and not have it buffer? Basically what I am seeing is that is writing, but it buffers records (this is in non-compressed form) ideally I'd like to have it write direct no buffering... is there a way to do this?

One of my parent directories had a space in it. This cause the make to fail.

gcc -Wall -Wno-deprecated-declarations -g -O3 -DDARWIN=1 -DVERSION="1.2" -o /Users/aradford1/Documents/work/interest/PROJECTS/UNSW - research/work/joy/bin/pcap2flow -I ./include -I /opt/local/include pcap2flow.c p2f.c config.c osdetect.c anon.c pkt_proc.c nfv9.c tls.c classify.c radix_trie.c hdr_dsc.c procwatch.c addr_attr.c addr.c wht.c http.c str_match.c acsm.c dns.c example.c -L /usr/lib -lpcap -lm -lcrypto -lz
clang: error: -E or -x required when input is from standard input
clang: error: no such file or directory: 'research/work/joy/bin/pcap2flow'
make[1]: *** [pcap2flow] Error 1
make: *** [pcap2flow] Error 2

I'm running OS X El Capitan (10.11.1), and I can't build cleanly.

My environment consists of:

• OS X 10.11.1
• Xcode 7.2 (and its command-line tools)
• clang Apple LLVM version 7.0.2 (clang-700.1.81)

On the first attempt, I get missing references to OpenSSL include headers and the system endian header. The former is because Apple deprecated the use of OpenSSL [1], and the latter seems to be a platform-specific location thing.

$ make
cd src; /Applications/Xcode.app/Contents/Developer/usr/bin/make
Makefile variables:
sysname=DARWIN
version=1.0
CWD=src
gcc -Wall -Wno-deprecated-declarations -g -O3   -mmacosx-version-min=10.5 -DDARWIN=1 -DVERSION=\"1.0\" -o pcap2flow  pcap2flow.c p2f.c config.c osdetect.c anon.c pkt_proc.c nfv9.c tls.c classify.c radix_trie.c hdr_dsc.c procwatch.c addr_attr.c addr.c wht.c -lpcap     -lm       -lcrypto
In file included from pcap2flow.c:69:
./anon.h:59:10: fatal error: 'openssl/aes.h' file not found
#include <openssl/aes.h>
         ^
1 error generated.
In file included from p2f.c:54:
./anon.h:59:10: fatal error: 'openssl/aes.h' file not found
#include <openssl/aes.h>
         ^
1 error generated.
In file included from config.c:42:
./anon.h:59:10: fatal error: 'openssl/aes.h' file not found
#include <openssl/aes.h>
         ^
1 error generated.
anon.c:58:10: fatal error: 'openssl/aes.h' file not found
#include <openssl/aes.h>
         ^
1 error generated.
In file included from pkt_proc.c:49:
./pkt.h:44:10: fatal error: 'endian.h' file not found
#include <endian.h>
         ^
1 error generated.
procwatch.c:73:10: fatal error: 'openssl/sha.h' file not found
#include <openssl/sha.h>
         ^
1 error generated.
make[1]: *** [pcap2flow] Error 1
make: *** [pcap2flow] Error 2

On my second attempt, I configured CFLAGS in src/Makefile to point to my local OpenSSL installation, but picked up two new errors: first one related to the inlining of wht_init (I think this might be compiler-specific), and a second one concerning the location of endian.h (I think this is platform-specific).

cd src; /Applications/Xcode.app/Contents/Developer/usr/bin/make
Makefile variables:
sysname=DARWIN
version=1.0
CWD=src
gcc -Wall -Wno-deprecated-declarations -g -O3   -I/usr/local/opt/openssl/include -mmacosx-version-min=10.5 -DDARWIN=1 -DVERSION=\"1.0\" -o pcap2flow  pcap2flow.c p2f.c config.c osdetect.c anon.c pkt_proc.c nfv9.c tls.c classify.c radix_trie.c hdr_dsc.c procwatch.c addr_attr.c addr.c wht.c -lpcap     -lm       -lcrypto
p2f.c:1118:66: warning: format specifies type 'long' but the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
  fprintf(output, "\t\t\t\"ts\": %ld.%06ld,\n", ts_start.tv_sec, ts_start.tv_usec);
                                     ~~~~~                       ^~~~~~~~~~~~~~~~
                                     %06d
p2f.c:1119:64: warning: format specifies type 'long' but the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
  fprintf(output, "\t\t\t\"te\": %ld.%06ld,\n", ts_end.tv_sec, ts_end.tv_usec);
                                     ~~~~~                     ^~~~~~~~~~~~~~
                                     %06d
In file included from p2f.c:52:
In file included from ./p2f.h:56:
./wht.h:53:13: warning: inline function 'wht_init' is not defined [-Wundefined-inline]
inline void wht_init(struct wht *wht);
            ^
p2f.c:423:3: note: used here
  wht_init(&record->wht);
  ^
3 warnings generated.
In file included from pkt_proc.c:49:
./pkt.h:44:10: fatal error: 'endian.h' file not found
#include <endian.h>
         ^
1 error generated.
make[1]: *** [pcap2flow] Error 1
make: *** [pcap2flow] Error 2

On my third attempt, I took the inline function from src/wht.c, and replaced the definition in src/wht.h with the function itself. I changed src/pkt.h to look for machine/endian.h.

I ran into a new error regarding radix_trie_node_add_flag_to_all_leaves:

cd src; /Applications/Xcode.app/Contents/Developer/usr/bin/make
Makefile variables:
sysname=DARWIN
version=1.0
CWD=src
gcc -Wall -Wno-deprecated-declarations -g -O3   -I/usr/local/opt/openssl/include -mmacosx-version-min=10.5 -DDARWIN=1 -DVERSION=\"1.0\" -o pcap2flow  pcap2flow.c p2f.c config.c osdetect.c anon.c pkt_proc.c nfv9.c tls.c classify.c radix_trie.c hdr_dsc.c procwatch.c addr_attr.c addr.c wht.c -lpcap     -lm       -lcrypto
p2f.c:1118:66: warning: format specifies type 'long' but the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
  fprintf(output, "\t\t\t\"ts\": %ld.%06ld,\n", ts_start.tv_sec, ts_start.tv_usec);
                                     ~~~~~                       ^~~~~~~~~~~~~~~~
                                     %06d
p2f.c:1119:64: warning: format specifies type 'long' but the argument has type '__darwin_suseconds_t' (aka 'int') [-Wformat]
  fprintf(output, "\t\t\t\"te\": %ld.%06ld,\n", ts_end.tv_sec, ts_end.tv_usec);
                                     ~~~~~                     ^~~~~~~~~~~~~~
                                     %06d
2 warnings generated.
Undefined symbols for architecture x86_64:
  "_radix_trie_node_add_flag_to_all_leaves", referenced from:
      _radix_trie_add_subnet in radix_trie-37d328.o
      _radix_trie_unit_test in radix_trie-37d328.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [pcap2flow] Error 1
make: *** [pcap2flow] Error 2

It seemed to be another inlining issue, but not quite. At this point, I'm stumped.

Hi, I have some problem with build joy on mac os (High Sierra version 10.13.3)

After I run ./config
and run make

Building libjoy.so ...
rm: /Users/oborysen/joy_project/joy1/lib/libjoy.so.3.0.0-dev.dSYM: is a directory
make[1]: *** [libjoy.so] Error 1
make: *** [all] Error 2

I am trying to use the scripts which are in analysis folder but there is no example files to test those commands which are described in README file.

I had downloaded and when i try to execute make command generates the following error. Please check

When running ifconfig, I get a list of interfaces that include lo0, eno0, docker0, lots of other interfaces, however, when I pick one, and try to run joy on it, I get "interface not found" (eno1). And then it lists the interfaces all as intf1-intf19 so there isn't a correlation between how linux shows the interfaces and joy accesses them. This seems odd from a programatic/administration perspective.

I.e. to guess which interface, I had to find the one with the proper IP. For some non-communications interfaces, they may not have IPs so how would I actually select the right one (except guess and check?)

I understand it has to be compiled in. But can we compile it in, and allow put a setting in the config to determine whether it's enabled or not? I didn't set the compressed out and am getting compressed output...