libtraceteam / libtrace Goto Github PK

Hi,

This may be a duplicate of #14, but for gzipped files.

I have a set of pcaps I'd like to parse and analyze with some custom code. Everything works fine when the pcaps are uncompressed. When I compress them (via gzip my.pcap) and re-run, libtrace errors out with the following message:

Error: RT type 20 cannot be converted to a pcap DLT
Assertion failed: (false), function rt_to_pcap_linktype, file linktypes.c, line 162.
./analysis.sh: line 5:  6042 Abort trap: 6           ./analysis "$FNAME"

Am I doing something wrong trying to run a gzipped file through the same code as an uncompressed file, or does libtrace not support reading in compressed pcaps? If it's the latter, I could possibly work on a PR to enable this option, but want to confirm it's not user error (me) first :)

Here's how I'm opening the file, it's pretty basic:

libtrace_t* file = trace_create(fpath.c_str());
libtrace_packet_t* packet = trace_create_packet();
libtrace_filter_t* bpf = trace_create_filter(bpf_s.c_str()); //bpf = udp

if(!file) throw std::runtime_error("Problem in trace create");
if(trace_config(file, TRACE_OPTION_FILTER, bpf) == -1) throw std::runtime_error("Problem in trace config");
if(trace_start(file) < 0) throw std::runtime_error("Problem in trace_start");

In tracesplit jump-to-IP-header mode, due to pointer updating and stale caching the reader always return TRACE_TYPE_NONE after the first packet instead of the real linktype causing incorrect conversion (or lack thereof as every packet is effectively treated as if it is already raw IP).

It appears that tracesplit jump updates the packet pointer to a new generated IPv4 output one, then the next libtrace_get_packet() call for some unknown reason checks the libtrace_t is the same in the packet structure before clearing out the old data. Because the new IPv4 packet has a new fake dummy trace_t attached to it the stale cache linktype remains and gets used for the input packet.

I don't see why tracesplit destroys the original input packet buffer it would normally re-use either, it should probably destroy the temporary one and continue using the real input packet; nor why per_packet passes in a libtrace_packet_t**.

Create new versions of all our trac wiki pages here on the GitHub wiki so we can eventually decommission the old trac wikipages and replace them with links to the GitHub wiki.

The main problem is the UserDocumentation, which contains 20+ wiki pages, so converting that will be tedious and time-consuming.

When converting the documentation across, it would be a very good idea to review it carefully and make sure it is actually correct for recent versions of libtrace.

Recently had a complaint about compilation failing on Ubuntu 16.04. I'll just paste the pertinent details here:

=======================================================

when I install the source code of libtrace after I type the commands:
./configure
Make
It generates me errors:
^
trace.c: In function 'trace_create_filter_from_bytecode':
trace.c:1358:8: error: 'struct libtrace_filter_t' has no member named 'filter'
filter->filter.bf_insns = (struct bpf_insn *)
^
trace.c:1359:17: error: invalid application of 'sizeof' to incomplete type 'struct bpf_insn'
malloc(sizeof(struct bpf_insn) * bf_len);
^
trace.c:1361:15: error: 'struct libtrace_filter_t' has no member named 'filter'
memcpy(filter->filter.bf_insns, bf_insns,
^
trace.c:1362:20: error: invalid application of 'sizeof' to incomplete type 'struct bpf_insn'
bf_len * sizeof(struct bpf_insn));
and many others errors like this.

======================================================
Definitely an issue with some missing definitions that we need for libtrace_filter_t.

I also got them to send me their config.h (attached). Libtrace is happy that they have libpcap installed, but pcap-bpf.h is missing. net/bpf.h is also not present (unsurprising, given they're on Linux).

Since we clearly need one or the other, we should try to detect this at the ./configure stage and prevent the build from going ahead (with a helpful error message pointing them at the library that they need to install, of course).

There are plenty of places in the core libtrace code where we are using assertions as an error / bug detection mechanism, i.e. if the assertion fails, then we need to pay some attention to that code.

This is good for us as developers, but not so great for our users when their programs crash just because something happened that we didn't anticipate.

Instead, we should be returning some sort of error code to allow users to handle the problem in their own way; maybe they want to carry on and ignore that packet, maybe they want to dump that packet out for further investigation, etc.

[the following was found on Debian Linux (Stretch)] Configuring a bpf filter results in a situation where the heap memory allocated for filter is orphaned inside the library and not accessible to be freed by the API. An excerpt from a valgrind session showing the call-chain is below:

==12792== 488 bytes in 2 blocks are definitely lost in loss record 1 of 1
==12792==    at 0x4C2DBC5: calloc (vg_replace_malloc.c:711)
==12792==    by 0x6479ED1: ??? (in /usr/lib/x86_64-linux-gnu/libpcap.so.1.8.1)
==12792==    by 0x646D0E2: pcap_compile (in /usr/lib/x86_64-linux-gnu/libpcap.so.1.8.1)
==12792==    by 0x586E150: linuxnative_configure_bpf (format_linux_common.c:104)
==12792==    by 0x586E150: linuxcommon_config_input (format_linux_common.c:150)
==12792==    by 0x58605A0: trace_config (trace.c:597)
...

This is easy to reproduce via a few small modifications to the example program “stats”:

1. Remove the infinite loop and run trace_read_packet() manually a handful of times
2. Add the missing call to trace_destroy_filter() near the end of the program.

Run with something like $ sudo ./stats --filter tcp ring:enp2s0f1 under valgrind with --leak-check=full to see something similar to above.

The leak is due to the fact that libtrace creates a local copy of the filter structure , configures that and then stores it internally in the opaque libtrace_t object (the internals of which are not directly accessible to the API). The end-user is instead left with the husk of what “filter” was before the copy — a dynamically-allocated sparely populated structure (created by trace_create_filter() (aside: when allocated, this should be calloc()’d or memset() to zero at inception because it currently contains a few uninitialized members). Calling trace_destroy_filter() using the filter created in trace_create_filter() will free filter->filterstring, but since filter->flag == 0, pcap_freecode() won't be called to release the memory allocated by pcap_compile() — but even if it did, it would be passing in the wrong filter structure (the one that was user-accessible, not the internal one held by the library). The result is that libtrace holds onto the memory for filter indefinitely. To fix, all of this could subsumed inside the library.

Special thanks to @shw700 for the second set of eyes.

I connected 2 workstations directly.
1 sends traffic with pktgen-dpdk and 1 receives with tracestats with threads.

A) When I press Ctrl+C, tracestats(8threads) doesn't stop. if I use 1 thread, trace_pstop() works well.

B) Another scenario is tracestats with no traffic.(absolutely no packet), trace_pstop() doesn't work.

If the supplied linktype does not have a corresponding DLT, trace_construct_packet() just barrels on with a DLT of -1, leading to an RT type of TRACE_RT_DATA_DLT-1 without error and blindly copying in whatever header and payload was in the packet without any conversion as if it was PCAP. tracereplay then dies later on when l3 conversion attempts to convert RT TRACE_RT_DATA_DLT-1=1999 to a sensible linktype.

Workaround: use tracesplit/traceconvert instead.

e.g.

$ tracereplay erf_meta_example_pkts.erf pcap:erf_meta_example_pkts_libtrace.pcap
Error: RT type 1999 cannot be converted to a pcap DLT
tracereplay: linktypes.c:162: rt_to_pcap_linktype: Assertion `false' failed.
Aborted (core dumped)

I'm interested in converting to the 4.0 API, has anyone already started on this ?

As reported by Mike Schiffman, trace_interrupt will only halt a pcapint: live capture, not ring:, int: or bpf:

To make this work for the other formats, we need to add a timeout to the blocking operation inside their read_packet function so we can periodically check the value of libtrace_halt. For ring: this should be easy -- add the timeout to the poll() call. For int: we might have to wrap a select() call around the recvmsg() call to act as a timeout. If there is a packet on the fd, we should break out of the select immediately. I imagine bpf: works in a similar way.

Will also need to look into extending this to DAG as well.

in each format_xxx.c, libtrace_halt is checked in timeout case.
but when traffic is busy, timeout is not occured so interrupt() doesn't work.
Easy fix would be move out libtrace_halt check to the first line in the read loop
but is there any intentional reason not to? (volatile int read performance? I don't know.)

Pcap files opened with trace_create() are not closed after trace_destroy(), which causes exhausting number of opened filedescriptors while trying to process more files in sequence.

when is the porting to DPDK-2.2 planned?
is anyone already working on this?

we're trying to get this working, so once we have something we'll be glad to contribute back.

Alex

Hello,

Using the latest git libtrace 4.0 and running the following:

tracesplit -Z gzip -i 600 dpdk:0000:03:00.0 pcapfile:10min_trace.pcap.gz

(no output or errors after the command line)

The output file is not comrpessed:

file 10min_trace.pcap.gz-1458152907.gz
10min_trace.pcap.gz-1458152907.gz: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 65536)

Any ideas ?

As of DPDK 1.7 drivers are loaded using a constructor attribute((constructor)), these are placed into the .init_array section of the DPDK static library.

However they appear to be lost when linking into libtrace.so.

One solution might be to link against their shared object. Otherwise figure out the linker flags needed.

Hi,

Have you seen this before it appears to be crashing in FORMAT_DATA_FIRST(libtrace)->queue_id

tracestats dpdk:0000:04:00.0
dpdk:0000:04:00.0:
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Detected lcore 4 as core 4 on socket 0
EAL: Detected lcore 5 as core 5 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 6 lcore(s)
EAL: Auto-detected process type: PRIMARY
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1c400000 bytes
EAL: Virtual area found at 0x7f83d9200000 (size = 0x1c400000)
EAL: Ask a virtual area of 0x7800000 bytes
EAL: Virtual area found at 0x7f83d1800000 (size = 0x7800000)
EAL: Ask a virtual area of 0x31800000 bytes
EAL: Virtual area found at 0x7f839fe00000 (size = 0x31800000)
EAL: Ask a virtual area of 0x3400000 bytes
EAL: Virtual area found at 0x7f839c800000 (size = 0x3400000)
EAL: Ask a virtual area of 0x1800000 bytes
EAL: Virtual area found at 0x7f839ae00000 (size = 0x1800000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f839a800000 (size = 0x400000)
EAL: Ask a virtual area of 0x38c400000 bytes
EAL: Virtual area found at 0x7f800e200000 (size = 0x38c400000)
EAL: Ask a virtual area of 0xc00000 bytes
EAL: Virtual area found at 0x7f800d400000 (size = 0xc00000)
EAL: Ask a virtual area of 0x800000 bytes
EAL: Virtual area found at 0x7f800ca00000 (size = 0x800000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f800c400000 (size = 0x400000)
EAL: Requesting 256 pages of size 2MB from socket 0
EAL: TSC frequency is ~2099998 KHz
EAL: Master lcore 0 is ready (tid=fe2b7a40;cpuset=[0])
EAL: lcore 5 is ready (tid=d89fe700;cpuset=[5])
EAL: lcore 3 is ready (tid=f5e45700;cpuset=[3])
EAL: lcore 2 is ready (tid=f6646700;cpuset=[2])
EAL: lcore 1 is ready (tid=f6e47700;cpuset=[1])
EAL: lcore 4 is ready (tid=d91ff700;cpuset=[4])
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: PCI memory mapped at 0x7f83fe182000
EAL: PCI memory mapped at 0x7f83fe2cd000
PMD: eth_igb_dev_init(): port_id 0 vendorID=0x8086 deviceID=0x1521
Intel DPDK setup
---Version : RTE 2.2.0
---Master LCore : 5
---LCore Count : 1
---Core 0 : off
---Core 1 : off
---Core 2 : off
---Core 3 : off
---Core 4 : off
---Core 5 : on
---Process Type : primary
Running pstart DPDK tot=1 req=1 phys=6
Creating mempool named libtrace_pool_6

Core 127 doing packet RX.
Segmentation fault (core dumped)

---------- GDB Output -------------------------------------------------------------

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/local/bin/tracestats...done.
[New LWP 19538]
[New LWP 19542]
[New LWP 19541]
[New LWP 19545]
[New LWP 19543]
[New LWP 19539]
[New LWP 19544]
[New LWP 19540]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `tracestats dpdk:0000:04:00.0'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f83fdbb361c in dpdk_start_input (libtrace=0x1c63300) at format_dpdk.c:1459

1459 FORMAT_DATA_FIRST(libtrace)->queue_id = rte_lcore_id();
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 glibc-2.17-106.el7_2.4.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.13.2-10.el7.x86_64 libcom_err-1.42.9-7.el7.x86_64 libcurl-7.29.0-25.el7.x86_64 libgcc-4.8.5-4.el7.x86_64 libibverbs-1.1.8mlnx1-OFED.3.2.1.5.0.32200.x86_64 libidn-1.28-4.el7.x86_64 libnl-1.1.4-3.el7.x86_64 libpcap-1.5.3-8.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 libssh2-1.4.3-10.el7.x86_64 libstdc++-4.8.5-4.el7.x86_64 lzo-2.06-8.el7.x86_64 nspr-4.10.8-2.el7_1.x86_64 nss-3.19.1-19.el7_2.x86_64 nss-softokn-freebl-3.16.2.3-13.el7_1.x86_64 nss-util-3.19.1-4.el7_1.x86_64 openldap-2.4.40-8.el7.x86_64 openssl-libs-1.0.1e-51.el7_2.4.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
(gdb) back
#0 0x00007f83fdbb361c in dpdk_start_input (libtrace=0x1c63300) at format_dpdk.c:1459
#1 0x00007f83fdba9353 in trace_pstart (libtrace=0x1c63300, global_blob=global_blob@entry=0x0, per_packet_cbs=per_packet_cbs@entry=0x1c654c0, reporter_cbs=reporter_cbs@entry=0x1c65520) at trace_parallel.c:1730
#2 0x000000000040133a in run_trace (threadcount=1, uri=0x7fffc613f5cf "dpdk:0000:04:00.0") at tracestats.c:209
#3 main (argc=2, argv=) at tracestats.c:281

I'm using RedHat 7.3,and modified the code to check rte_lcore_id() works and its returning 127

Core 127 doing packet RX.

In single thread scenario,
libtrace_message_queue_init(hasher_thread.messages) is not called, but
libtrace_message_queue_destroy(hasher_thread.messages) is always called.

As result, hasher_thread.message.pipe fd is closed even though it has garbages.
(My case the garbage was pipe[0]=0, pipe[1]=1, so stdout was closed accidently.)

tested on libtrace 4.0.2 but I think it is same in latest build.

Similar to Issue #47, we had an IPv6 packet encapsulated in an IP packet, quoted in an ICMP packet, but the IPv6 parsing would go a little crazy:

Mon Apr 17 16:31:43 2017
 Capture: Packet Length: 98/102 Direction Value: -1
 Ethernet: Dest: xx Source: xx Ethertype: 0x0800
 IP: Header Len 20 Ver 4 DSCP 00 ECN 0 Total Length 84
 IP: Id 0 Fragoff 0
 IP: TTL 241 Proto 1 (icmp) Checksum 49013
 IP: Source xx Destination xx
 ICMP: Type: 3 (ICMP Destination Unreachable)
 ICMP: Code: 4 (Fragmentation Required And Dont Fragment Set)
 ICMP: Checksum: 38984
 IP: Header Len 24 Ver 6 DSCP 1f ECN 0 Total Length 29861
 IP: Id 47964 Fragoff 4355
 IP: TTL 139 Proto 41 (ipv6) Checksum 56308
 IP: Source xx Destination xx
 IPv6: Version 13
 IPv6: Class 31
 IPv6: Flow Label 161804
 IPv6: Payload Length 20163
 IPv6: Next Header 0
 IPv6: Hop Limit 60
 IPv6: Source IP xx
 IPv6: Destination IP xx
 IPv6 Hop-by-Hop: Next Header 0 Header Ext Len 0
 IPv6 Hop-by-Hop: Next Header 0 Header Ext Len 0
 IPv6 Hop-by-Hop: Next Header 0 Header Ext Len 0
 IPv6 Hop-by-Hop: Next Header 0 Header Ext Len 0
 IPv6 Hop-by-Hop: Next Header 0 Header Ext Len 0
 IPv6 Hop-by-Hop: Next Header 0 Header Ext Len 0
....

I've patched this by checking that the length is long enough to at least hold the v6 header:

diff --git a/libpacketdump/eth_34525.c b/libpacketdump/eth_34525.c
index c5e549f..bf78764 100644
--- a/libpacketdump/eth_34525.c
+++ b/libpacketdump/eth_34525.c
@@ -36,6 +36,10 @@

 DLLEXPORT void decode(int link_type UNUSED,const char *packet,unsigned len)
 {
+       if (len < sizeof(libtrace_ip6_t)) {
+               /* TODO: figure out which fields could be read safely? */
+               return;
+       }
        libtrace_ip6_t *ip = (libtrace_ip6_t*)packet;

        uint32_t tmp = ntohl(*(uint32_t*)ip);

Do you have a better idea?

In format_dag25.c:617 we clear any ERF records before the trace started. After this process we have a top pointer and a bottom pointer pointing in the DAG memory map.

Shane made changes so that any packets received during this process are kept so we can read them (meaning the trace starts as soon as we ask it to start). However, there is an error when the memory is already clear. The value to 'top' depends on there being memory to clear.

starttop = dag_advance_stream(FORMAT_DATA->device->fd,
                                        FORMAT_DATA->dagstream,
                                        &bottom);

/* Should probably flush the memory hole now */
while (starttop - bottom > 0) {
    bottom += (starttop - bottom);
    top = dag_advance_stream(FORMAT_DATA->device->fd,
                FORMAT_DATA->dagstream,
                &bottom);
}
FORMAT_DATA->top = top;
FORMAT_DATA->bottom = bottom;

If we set the top value before the loop, this issue will be fixed

starttop = dag_advance_stream(FORMAT_DATA->device->fd,
                                        FORMAT_DATA->dagstream,
                                        &bottom);

/* Should probably flush the memory hole now */
top = starttop;
while (starttop - bottom > 0) {
    bottom += (starttop - bottom);
    top = dag_advance_stream(FORMAT_DATA->device->fd,
                FORMAT_DATA->dagstream,
                &bottom);
}
FORMAT_DATA->top = top;
FORMAT_DATA->bottom = bottom;

Staring line lib/format_dat25.c:449 we have the 'dag_init_input' method which will process the input URI and open the DAG device for capture. It makes use of strdup to copy the device name from the URI (presumably to get null termination on the end without destroying the rest of the URI).

In the case the device is already open or the device fails to open this memory will be freed. However in the typical case where a device gets opened, the memory will leak.

We can fix this by adding a free at the end of the method as we don't actually need the device name after we have initialised. This isn't strictly true (as I need to configure the card in dag_config_input, which requires reference to the card name), but it's easy enough to parse it from the URI if we ever need to.

I downloaded develop branch and run
'examples/skeleton/parallel eth1' and 'Ctrl+C' then error occurs.

lt-parallel: trace_parallel.c:808: hasher_entry: Assertion `trace->state == STATE_FINISHED' failed.
Abort (core dumped)

sometimes another error
Processing packest(eth1): You must call libtrace_start() before trace_read_packet()

Redhat 6.7
eth1 has very low or no traffic
2017-04-21 version(latest commit is f398c61)

Pasted from an email I received:

I recently upgraded Endace to latest version and now, Endace, has included provenance records in its capture and it has broken your library. libtrace will not recognize the erf capture format. Do you guys plan on supporting these new records?

I recently changed my application to use libtrace.a instead of libtrace.so.
but then I had to link libwandio.a and dpdk.a as well.

I thought libtrace.a includes all symbols from libwandio.a and libintel_dpdk.a.
$ nm libtrace.a | grep wandio -> shows 'U' not 'T'
$ nm libtrace.a | grep rte -> shows 'U' not 'T'

Is it usual for static library?
I'm not familiar with static link so please correct me if I'm wrong.

TLDR: Store packet information (timestamp, direction, size etc.) in the libtrace packet header rather than in the format's packet header, otherwise we are relying on format support.

Libtrace formats all support different fields with some being more restrictive than others, a libtrace packet is tied to a specific format. Calling the set and get functions on a packet call the format specific functions which store this information in the packet format header (included in the packet buffer).

Packets read from a format without a field cannot be set and then stored to a format with support. The results in odd behaviour, for example running:
tracemerge --set-interface erf:output.erf erf:in1.erf erf:in2.erf
Will result in a unique direction/port assigned to in1.erf and in2.erf in the merged result output.erf.
However running the same command with pcap files as input and erf as output will not, despite the erf format supporting the direction field.

Most of these fields are already cached in the libtrace packet, so this would not drastically increase its size. Logic would need to be updated to allow writing to these without format support, and deal with cases such as the format only providing a get and not a set.

It might be best to pre-emptively fill all these cached values when a packet is read, I suspect this would end up faster. When getting a field this would remove the need to check for and then call the format specific get option. Another consideration would be ensuring that setting a field gets written to the format header, so that writing back to the same format type wont lose the updated field. I'm certain there will be more edge cases here.

Other advantages include:
As a result user constructed packets no longer would need to be tied to a 'dead format'.
The resulting packet could also then be used as the rt header.

Libtrace will now attempt to recognise that a packet may point to memory that is no longer valid, as its parent trace has been restarted since the packet was read.

Some ideas for test cases that we should add to the test suite:

• check that all packets read in a scenario where a trace is never restarted do not have a link type of CONTENT_INVALID.
• check that CONTENT_INVALID is returned by trace_get_link_type for packets after the trace is restarted.
• check that NULL is returned by trace_get_packet_buffer (and other layer access functions) after the trace is restarted.
• check that capture length, wire length, timestamp, etc. access functions return appropriate error codes after the trace is restarted.
• check that any packets read after restarting are not marked as CONTENT_INVALID.
• check that CONTENT_INVALID then applies to the second batch of packets read following a second restart.

Feel free to add more test ideas to this issue.

We ran into a problem when dumping an SCTP packet that is quoted inside an ICMP dest unreach packet where the SCTP parser would blindly believe the quoted chunk length with no regard to the overall packet length passed into the decode function.

We'd get infinite output that looks like this:

Mon Apr 17 15:45:29 2017
 Capture: Packet Length: 122/126 Direction Value: -1
 Ethernet: Dest: xx Source: xx Ethertype: 0x0800
 IP: Header Len 20 Ver 4 DSCP 00 ECN 0 Total Length 108
 IP: Id 0 Fragoff 0
 IP: TTL 241 Proto 1 (icmp) Checksum 65523
 IP: Source xx Destination xx
 ICMP: Type: 3 (ICMP Destination Unreachable)
 ICMP: Code: 4 (Fragmentation Required And Dont Fragment Set)
 ICMP: Checksum: 10633
 IP: Header Len 48 Ver 1 DSCP 00 ECN 2 Total Length 37414
 IP: Id 59048 Fragoff 1372 MORE_FRAG DONT_FRAG
 IP: TTL 13 Proto 132 (sctp) Checksum 6909
 IP: Source xx Destination xx
 SCTP: Header Src port 30086 Dst port 17631 Tag 1992624677 Csum 3637884784
 SCTP: Chunk 1 Type INIT ACK Flags 54 Len 51751
 SCTP: Tag 764166009 Credit 2396187391 Outbound 19912 Inbound 38008 TSN 2081719691
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0
 SCTP: Option Unknown type=0 len=0

Note the line

 SCTP: Chunk 1 Type INIT ACK Flags 54 Len 51751

That length (51751) is far larger than the actual packet length, but the SCTP parser happily uses it to try and parse the options (https://github.com/LibtraceTeam/libtrace/blob/master/libpacketdump/ip_132.c#L244)

I quickly patched this by preferring the smaller of vlen and len:

diff --git a/libpacketdump/ip_132.c b/libpacketdump/ip_132.c
index 9889161..103917d 100644
--- a/libpacketdump/ip_132.c
+++ b/libpacketdump/ip_132.c
@@ -237,7 +237,12 @@ DLLEXPORT void decode(int link_type UNUSED,const char *packet,unsigned len)
                         ntohs(ack->inbound_streams),
                         ntohl(ack->init_tsn));

-                vlen = chunk->length - (sizeof(struct sctp_init_ack) +
+               if (chunk->length > len) {
+                       vlen = len;
+               } else {
+                       vlen = chunk->length;
+               }
+                vlen = vlen - (sizeof(struct sctp_init_ack) +
                         sizeof(struct sctp_chunk_hdr) +
                         sizeof(struct sctp_common_hdr)
                         );

But maybe this isn't the correct fix.
Thoughts?

We can write LZO traces but we don't provide any mechanism for reading them back in again. Seeing as it might become more common for people to write LZO traces in future to keep up with high line rates, it would be nice for libtrace to be able to read them natively.

At the moment, this is a hard-coded constant value of 10 which means any transmits tend to happen in batches of 10 packets. This is helpful for increasing performance when sending large volumes of traffic, but is much less helpful when the inter-packet timing needs to be preserved or the packet must be sent urgently (e.g. tracereplay).

If configurable, tracereplay could change this to a more suitable value (such as 1), whereas high throughput applications could continue to use 10 (or some larger value?).

10 is still probably a good default, so we should use that if no alternative configuration is provided.

I'm using libtrace develop 6d17620 on centos 7.0.
When I run 'tracestats pcapint:eno1' and press ctrl+c,
program doesn't end.
It was hanging at first pthread_mutex_lock(&libtrace->libtrace_lock) in trace_ppause().
int:eno1, ring:eno1 works well, though.

Does the LibtraceTeam maintain a git repository for libwandio? It is a dependency that is very closely tied to libtrace, but I can't find an official repository for it, or if any development on wandio has occurred in the last 2 years.

When running tracertstats -i 1 on FreeBSD with more than one processing thread, output tends to appear in bursts of several lines appearing at multiple second intervals rather than the one line per second that would be expected.

Attaching truss to the running process seems to make the program behave correctly again, so it is a bit hard to get visibility into what is going on.

Program also runs fine if only one processing thread is used (i.e. -t 1 on the CLI).

Adding a specific "fflush(stdout);" to the output function doesn't change anything, so it doesn't appear to be a stdio buffering issue.

Newer trace formats have support for including meta-data in the packet stream, either as separate records (e.g. ERF provenance) or as part of the packet record itself (e.g. pcap-ng).

We need to think about a relatively format-agnostic way of allowing people to interact with this meta-data, both in terms of reading it and creating it. The meta-data is typically quite flexible, so we have to allow for a variety of custom meta-data fields and values, in addition to the standard meta-data records such as those produced by Wireshark or the Endace Probe software.

Any idea what I might be missing?

./bootstrap.sh
+ aclocal-1.11 -I m4
./bootstrap.sh: line 5: aclocal-1.11: command not found
+ aclocal-1.9 -I m4
./bootstrap.sh: line 6: aclocal-1.9: command not found
+ aclocal -I m4
aclocal: warning: autoconf input should be named 'configure.ac', not 'configure.in'
+ libtoolize --force --copy
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
+ autoheader2.50
./bootstrap.sh: line 13: autoheader2.50: command not found
+ autoheader
+ automake-1.11 --add-missing --copy --foreign
./bootstrap.sh: line 16: automake-1.11: command not found
+ automake-1.10 --add-missing --copy --foreign
./bootstrap.sh: line 17: automake-1.10: command not found
+ automake-1.9 --add-missing --copy --foreign
./bootstrap.sh: line 18: automake-1.9: command not found
+ automake --add-missing --copy --foreign
automake: warning: autoconf input should be named 'configure.ac', not 'configure.in'
configure.in:67: installing './compile'
configure.in:21: installing './missing'
examples/parallel/Makefile.am: installing './depcomp'
automake: warning: autoconf input should be named 'configure.ac', not 'configure.in'
+ autoconf2.50
./bootstrap.sh: line 21: autoconf2.50: command not found
+ autoconf

In format_pcapfile.c:355 we have:

        err=wandio_read(libtrace->io,
                        packet->buffer,
                        sizeof(libtrace_pcapfile_pkt_hdr_t));
        if (err<0) {
                trace_set_err(libtrace,errno,"reading packet");
                return -1;
        }
        if (err==0) {
                /* EOF */
                return 0;
        }

The problem is that wandio_read can return an error without errno being set (or at least that would explain what I'm seeing). Inside of trace_set_err there is an assert that the errcode is non-zero. This turns an error that I'd like to recover from into an assertion failure, which I cannot recover from.

Is there some way we can parallelise BPF, much like we already are able to do for ring? At present, BPF isn't the greatest when used with the parallel API so it would be nice if we could at least look into making life better for our BSD users.

I found libtrace very useful. Thank you guys.

Some getter functions accepts buffer and return the same.
So I just ignored the return value... then the logic didn't work.
For example,

  trace_get_source_address_string(packet, buf, len) // A
  printf("A buf=%s\n", buf);  // doesn't work.
  char* p = trace_get_source_address_string(packet, buf, len) // B
  printf("B buf=%s\n", buf);  // works fine.

This is because trace_get_source_address_string is declared with the compiler optimization attribute((pure)), which optimizes out the function if it doesn't use any global variables.
Compiler removes the first line because I didn't use return value.
(I used gcc 4.4.7 20120313 with no option.)

I'm interested in adding zstd support to libtrace. Before I start:

• 1. Would there be any interest in zstd?
• 2. What would be the best approach for adding zstd support to libtrace?

I build wandio 4.0.0 locally and try to build libtrace like this

$ export WANDIO_DIR=/my/wandio/build_dir
$ ./configure --without-ncurses CFLAGS="-L${WANDIO_DIR}/.libs -I${WANDIO_DIR}" LDFLAGS="-L${WANDIO_DIR}/.libs" --with-dpdk
then configure succeeded but make failed.
CXXLD libtrace.la
/usr/bin/ld: cannot find -lwandio

if I don't use --with-dpdk, there's no problem.
Any idea? or do I have to make install wandio to /usr/lib64

tested with libtrace 4.0.4, 4.0.5 on RHEL7

I've come across an interesting 'feature' of trace_event that causes the accepted_packets counter to be exactly twice as large as the real number of packets.

From what I can tell, this is because several formats (linux, pcap, bpf, tsh, pcapfile (when not realtime), legacy) call the trace_event_device and trace_event_trace functions in format_helper.c. These functions then call trace_read_packet which increments the accepted_packets counter (trace.c:745) (so far so good). However, when the format's trace_event function returns to trace_event in libtrace.c, the counter is incremented again (trace.c:1116).

I'm not submitting a PR because I am really not sure the 'best' way to fix this.
We can't simply remove the increment from trace_event because any formats that don't (directly or indirectly) increment the counter themselves will end up with a count of 0.
trace_event_device and trace_event_trace could decrement the counter after the call to trace_read_packet but this seems icky.

Thoughts?

P.S. the list of affected formats is based on my doing a quick grep through the code. I have only verified this behavior on the BPF format (and i have a year of data that corroborates my story ;) ).

I'm using X710 (i40e) and testing libtrace 3.22 latest version.

The rx traffic is 8Gbps, but libtrace 3.22 with dpdk only handles about 70% packets.
When I use dpdk pktgen to check rx performance, there is no packet drop.

So I look into the format_dpdk.c and found the difference in queue_setup.
libtrace use rx_conf = {pth=8,hth=8,wth=4,freeth=0} but
pktgen uses NULL, which means default rx_conf={8,8,0,32}. (from i40e_ethdev.h)

After I changed queue_setup(...,&rx_conf,...) to queue_setup(...,NULL,...),
there is no drop any more.

Unless there is any other reason, using default for rx/tx queue setup seems better.

Tested on dpdk 1.8.0 and dpdk 2.0.0.
(It seems libtrace is working with dpdk 2.0.0, Hooray~)

If you have a multi-threaded program where two threads both attempt to create a trace (alive or dead) at the same time, it is possible for a race condition to occur on the formats_list variable. This usually results in an assertion failure (f->next == NULL) in register_format().

It might be that we need to push the responsibility for calling trace_init() onto the user directly and hope that they call it in a sensible place. Another option might be to have one format list per trace object; this is a bit wasteful but we could perhaps mitigate this somewhat by only adding formats that are being actively used.

Note that conventional parallel libtrace programs are not affected by this issue, since they should always be creating a trace object in their main thread only, so there shouldn't be any opportunity for a race.

http://www.winpcap.org/ntar/draft/PCAP-DumpFileFormat.html

Starting to come across more and more trace files that use pcap-ng, so we should at least be able to read them.

Reported by rofl0r:

When doing make -j9 (using 8 parallel cores to build) the build will fail trying to compile wandiocat and maybe others that depend on libwandio - because libwandio was not created yet.
this could be fixed by making libwandio.la a dependency of wandiocat (and possibly other tools requiring the lib).

/usr/include is not a safe path to store files.

Cf. https://bugs.gentoo.org/show_bug.cgi?id=604660

DAG transmit in format_dag25 is somewhat broken:

• Records should be aligned to 64-bit with 0 padding or they will not transmit on physical cards. If the record is snapped it should be rounded down to the next lowest 8 byte boundary. Libtrace format_erf currently always strips padding when writing to ERF which effectively means no ERF file created by libtrace is transmittable by libtrace DAG.
• Extension headers are not copied correctly.
• The write functions of format_dag25 and format_erf should probably be unified.

As noted in #63, libpacketdump is currently unable to properly parse ERF provenance records. At the very least, we should be able to decode a "typical" provenance record such as those produced by the Endace Probe during normal operation.

I want to calc some stats and found clearing stat api is missing. memset can't help because of magic field. I think the field is kind of insurance to protect from corruption, but that requires to create heap even if all I need is local var in stack.

The provided solution for visual studio is not up to date.

• there are c and h files missing
• invalid (non ASCII) characters in c files
• absolute path to other projects like "C:\Users\root\Documents\Visual Studio 2005\Projects\WpdPack\Include"

I'm using Visual Studio 2015 Update 1 and moved config-win.h to lib/config.h as stated in WIN32.txt.

because it uses string buffer char str[20].
str[40] should do.

I want to fix this by commit & pull request myself but it is not easy in my office situation.
sorry about that.