GithubHelp home page GithubHelp logo

msantos / pkt Goto Github PK

View Code? Open in Web Editor NEW
151.0 23.0 44.0 223 KB

Erlang network protocol library

Home Page: http://blog.listincomprehension.com/search/label/epcap

License: BSD 3-Clause "New" or "Revised" License

Makefile 0.09% Erlang 99.91%
arp ethernet gre ipv4 ipv6 tcp udp dlt icmp icmpv6

pkt's Introduction

An Erlang network protocol library.

Originally part of epcap: http://github.com/msantos/epcap

EXPORTS

pkt:decapsulate(Data) -> Packet
pkt:decapsulate(Proto, Data) -> Packet

    Types   Data = binary()
            Proto = atom() | integer()
            Packet = [ Header | Payload ]
            Header = #ether{} | #arp{} | #null{} | #linux_cooked{} |
                #ipv4{} | #ipv6{} | #tcp{} | #udp{} | #sctp{} | #icmp{} |
                #icmp6{} | #igmp{} | #gre{} | #llc{} | #vrrp{} | #'802.1q'{} |
                #lldp{} | #mpls{} | #rarp{} | #'802.1x'{}
            Payload = binary()

    Convert network protocols from binary data to a list of Erlang
    records followed by the payload.

    decapsulate/1,2 works on valid packets. If the packet is malformed
    or unsupported, decapsulate/1 will crash.

    decapsulate/1 parses the data as an ethernet frame.

    decapsulate/2 allows specifying the protocol for decoding the
    packet. If the protocol is specified as an integer, the integer
    is treated as a datalink type.

pkt:decode(Data) -> {ok, Packet} | {error, SoFar, {FailedProto, binary()}}
pkt:decode(Proto, Data) -> {ok, Packet} | {error, SoFar, {FailedProto, binary()}}

    Types   Data = binary()
            Proto = FailedProto = atom()
            Packet = {Headers, Payload}
            Headers = [Header]
            Header = #ether{} | #arp{} | #null{} | #linux_cooked{} |
                #ipv4{} | #ipv6{} | #tcp{} | #udp{} | #sctp{} | #icmp{} |
                #icmp6{} | #igmp{} | #gre{} | #llc{} | #vrrp{} | #'802.1q'{} |
                #lldp{} | #mpls{} | #rarp{} | #'802.1x'{}
            SoFar = Headers | []
            Payload = binary()

    Similar to decapsulate/1 but, on error, returns any part of the
    packet that has been successfully converted to Erlang term format.

The following functions create the protocol headers, converting between records and binaries. See include/pkt.hrl for definition of the record types.

ether(Packet) -> {#ether{}, Payload} | binary()
llc(Packet) -> {#llc{}, Payload} | binary()
vrrp(Packet) -> {#vrrp{}, Payload} | binary()
'802.1q'(Packet) -> {#'802.1q'{}, Payload} | binary()
null(Packet) -> {#null{}, Payload} | binary()
linux_cooked(Packet) -> {#linux_cooked{}, Payload} | binary()
gre(Packet) -> {#gre{}, Payload} | binary()
arp(Packet) -> {#arp{}, Payload} | binary()
ipv4(Packet) -> {#ipv4{}, Payload} | binary()
ipv6(Packet) -> {#ipv6{}, Payload} | binary()
tcp(Packet) -> {#tcp{}, Payload} | binary()
sctp(Packet) -> {#sctp{}, Payload} | binary()
udp(Packet) -> {#udp{}, Payload} | binary()
icmp(Packet) -> {#icmp{}, Payload} | binary()
icmp6(Packet) -> {#icmp6{}, Payload} | binary()
igmp(Packet) -> {#igmp{}, Payload} | binary()
mpls(Packet) -> {#mpls{}, Next, Payload}
lldp(Packet) -> {#lldp{}, Payload}
rarp(Packet) -> {#rarp{}, Payload}
'802.1x'(Packet) -> {#'802.1x'{}, Payload}

    Types   Packet = Header | binary()
            Header = #ether{} | #null{} | #linux_cooked{} | #arp{} |
                #ipv4{} | #ipv6{} | #tcp{} | #sctp{} | #udp{} |
                #icmp{} | #icmp6{} | #igmp{} | #gre{} | #llc{} |
                #vrrp{} | #'802.1q'{} | #lldp{} | #mpls{} | #rarp{} | #'802.1x'{}
            Next = ipv4 | ipv6


makesum(Packet) -> integer()

    Types   Packet = IPv4_header | [IPv4_header, Header, Payload]
            IPv4_header = #ipv4{}
            Header = #tcp{} | #udp{}
            Payload = binary()

    Calculate the one's complement checksum of the packet.

    When computing the checksum, the header sum field must be set
    to 0:

        Sum = pkt:makesum([IPv4, TCP#tcp{sum = 0}, Payload]).

    For verifcation, the checksum can be compared to the value in
    the header or:

        0 = pkt:makesum([IPv4, TCP, Payload]).

EXAMPLES

  • decode an ethernet frame, displaying the source and destination of valid packets
Frame = <<224,105,149,59,163,24,0,22,182,181,62,198,8,0,69,0,0,54,2,108,64,
          0,53,6,172,243,173,192,82,195,192,168,213,54,0,80,143,166,75,154,
          212,181,116,33,53,92,128,24,0,126,60,199,0,0,1,1,8,10,92,104,96,
          16,22,69,237,136,137,0>>,

try pkt:decapsulate(Frame) of
    [#ether{}, #ipv4{saddr = Saddr, daddr = Daddr},
        #tcp{sport = Sport, dport = Dport}, _Payload] ->
        {{Saddr, Sport}, {Daddr, Dport}}
catch
    error:_ ->
        ok % ignore invalid packets
end
  • verify the TCP checksum of an ethernet frame
{ok, [#ether{}, IPv4, #tcp{sum = Sum} = TCP, Payload]} = pkt:decode(ether, Frame),

% Re-calculate the checksum and match against the checksum in the header
Sum = pkt:makesum([IPv4, TCP#tcp{sum = 0}, Payload]),

% Or just verify the checksum
0 = pkt:makesum([IPv4, TCP, Payload]).

PADDING OF ETHERNET FRAMES

The minimum size of an ethernet payload is 46 bytes. An ethernet frame containing a TCP/IP packet composed of a 20 byte IPv4 header and 20 byte TCP header and payload will be padded by 6 bytes. To calculate the actual length of the packet, use the length and header length values in the IPv4 header and the offset value in the TCP header:

[#ether{}, #ipv4{len = Len, hl = HL}, #tcp{off = Off}, Payload] = pkt:decapsulate(Frame),
    Size = Len - (HL * 4) - (Off * 4),
    <<Payload:Size/bytes>>.

TODO

  • support RFC 2675 (IPv6 Jumbograms)

  • IPv6 AH and ESP

    • handle alignment differences between IPv4 and IPv6 (IPv4 uses 32 bits, IPv6 uses 64 bits)
  • ICMPv6

    • fix handling of neighbour discovery
    • simplify ICMPv6 header record and add a record for ICMPv6 type or add functions for ICMPv6 variable length payloads
  • IGMP

    • support IGMPv3 variable payloads
  • merge in DLT_IEEE802_11 support from wierl

  • merge in ICMPv6 code from gen_icmp

  • DLTs

    • DLT_SLIP
    • DLT_PPP
    • DLT_RAW
    • DLT_PPP_SERIAL
    • DLT_PPP_ETHER
    • DLT_LOOP

pkt's People

Contributors

alexshavelev avatar amolenaar avatar ates avatar josemic avatar laf0rge avatar lefan avatar massemanet avatar maxlapshin avatar msantos avatar ned-jm avatar sebastiw avatar shun159 avatar systra avatar vasu-dasari avatar wrachwal avatar yannayl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pkt's Issues

Support for 0x886c

I was running epcap inside an Elixir program on a Raspberry PI 3 when I run into this error

** (FunctionClauseError) no function clause matching in :pkt_ether.type/1
    src/pkt_ether.erl:37: :pkt_ether.type(34924)
    src/pkt.erl:120: :pkt.decapsulate_next/2

34924 is 0x886c.
I googled and found https://www.raspberrypi.org/forums/viewtopic.php?t=146246&p=963226
If I understood that correctly, it's the builtin wifi card of the PI that is sending those packets.
I don't know what a LARQ is and I don't really now what I've been doing, but this fixed that:

diff --git a/include/pkt_ether.hrl b/include/pkt_ether.hrl
index 2412165..71107ba 100644
--- a/include/pkt_ether.hrl
+++ b/include/pkt_ether.hrl
@@ -7,6 +7,7 @@
 -define(ETH_P_LLDP, 16#88CC).
 -define(ETH_P_MPLS_UNI, 16#8847).
 -define(ETH_P_MPLS_MULTI, 16#8848).
+-define(ETH_P_LARQ, 16#886C).
 
 -record(ether, {
     dhost = <<0,0,0,0,0,0>> :: <<_:48>>,
diff --git a/src/pkt.erl b/src/pkt.erl
index a75519a..05e28df 100644
--- a/src/pkt.erl
+++ b/src/pkt.erl
@@ -191,7 +191,10 @@ decapsulate_next({vrrp, Data}, Headers) ->
     lists:reverse([Payload, Header | Headers]);
 % IPv6 NONE pseudo-header
 decapsulate_next({ipv6_none, Data}, Headers) ->
-    lists:reverse([Data|Headers]).
+    lists:reverse([Data|Headers]);
+% LARQ
+decapsulate_next({larq, _}, Headers) ->
+    Headers.
 
 decode(Data) when is_binary(Data) ->
     decode(ether, Data).
diff --git a/src/pkt_ether.erl b/src/pkt_ether.erl
index c00a955..512d063 100644
--- a/src/pkt_ether.erl
+++ b/src/pkt_ether.erl
@@ -46,7 +46,9 @@ type(?ETH_P_802_1Q) -> '802.1q';
 type(?ETH_P_802_1QinQ) -> '802.1qinq';
 %% MPLS_
 type(?ETH_P_MPLS_UNI) -> mpls;
-type(?ETH_P_MPLS_MULTI) -> mpls.
+type(?ETH_P_MPLS_MULTI) -> mpls;
+%% LARQ
+type(?ETH_P_LARQ) -> larq.
 
 codec(<<Dhost:6/bytes, Shost:6/bytes, Type:16, Payload/binary>>) ->
 %    Len = byte_size(Packet) - 4,

It's probably a hack but I hope it helps adding proper support for these packets.

decapsulate now returns something like

[{:ether, <<184, 39, 235, 46, 32, 156>>, <<186, 39, 235, 46, 32, 156>>, 34924, 0}]

Maybe it's not the Erlang way, but it could return unknown instead of failing when it doesn't know how to decapsulate something. The client would fail anyway if it doesn't handle that, or could decide to keep going on.

SCTP chunk's padding value is not zero

Hello,

I've faced a strange situation when chunks's padding value was not filled with zeros, as it is stated in RFC. Here is an example of SCTP packet:

<<11,89,11,89,109,75,153,162,11,83,232,167,5,0,0,26,0,1,0,22,150,192,147,82,62,
  25,29,131,1,0,120,136,16,85,0,64,0,0,1,1>>

The last two bytes are "padding" and looks like it's incorrect behaviour to have value of padding different from zero.

Has someone seen anything like that before?

Please do not close this issue for some time. It's easy to fix that in pkt_sctp module, but I want to wait for someone who can share some knowledge about that.

How to force ignore unknown ethertype

A huawei switch in my LAN broad cast packets whose ethernet type is 0x9998(39320). Then the sample sniff in epcap just crashed with the following stack. I change the field crash in the state to false but still no luck.

4> 2020-07-21T14:02:53.120415+08:00 error: State machine sniff terminating. Reason: function_clause. Stack: [{pkt_ether,type,[39320],[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt_ether.erl"},{line,37}]},{pkt,decapsulate_next,2,[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt.erl"},{line,120}]},{sniff,handle_info,3,[{file,"examples/sniff.erl"},{line,76}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1166}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]. Last event: {info,{packet,1,{1595,311372,700224},60,<<255,255,255,255,255,255,160,8,111,227,46,64,153,152,0,1,0,0,0,5,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}}. State: {sniffing,{state,<0.149.0>,true,[]}}.
2020-07-21T14:02:53.125811+08:00 error: crasher: initial call: sniff:init/1, pid: <0.146.0>, registered_name: sniff, error: {function_clause,[{pkt_ether,type,[39320],[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt_ether.erl"},{line,37}]},{pkt,decapsulate_next,2,[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt.erl"},{line,120}]},{sniff,handle_info,3,[{file,"examples/sniff.erl"},{line,76}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1166}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}, ancestors: [<0.139.0>], message_queue_len: 2, messages: [{packet,1,{1595,311372,938644},60,<<255,255,255,255,255,255,16,144,39,233,250,177,8,6,0,1,8,0,6,4,0,1,16,144,39,233,250,177,192,168,88,34,0,0,0,0,0,0,192,168,88,182,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>},{packet,1,{1595,311372,942218},60,<<255,255,255,255,255,255,16,144,39,233,250,177,8,6,0,1,8,0,6,4,0,1,16,144,39,233,250,177,192,168,88,34,0,0,0,0,0,0,192,168,88,181,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0>>}], links: [<0.149.0>], dictionary: [], trap_exit: true, status: running, heap_size: 10958, stack_size: 28, reductions: 9345; neighbours:
2020-07-21T14:02:53.126505+08:00 error: Generic server <0.149.0> terminating. Reason: {function_clause,[{pkt_ether,type,[39320],[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt_ether.erl"},{line,37}]},{pkt,decapsulate_next,2,[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt.erl"},{line,120}]},{sniff,handle_info,3,[{file,"examples/sniff.erl"},{line,76}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1166}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}. Last message: {'EXIT',<0.146.0>,{function_clause,[{pkt_ether,type,[39320],[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt_ether.erl"},{line,37}]},{pkt,decapsulate_next,2,[{file,"/Users/sheltont/bde/github/epcap/_build/default/lib/pkt/src/pkt.erl"},{line,120}]},{sniff,handle_info,3,[{file,"examples/sniff.erl"},{line,76}]},{gen_statem,loop_state_callback,11,[{file,"gen_statem.erl"},{line,1166}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}}. State: {state,<0.146.0>,#Port<0.15>}.

Screen Shot 2020-07-21 at 2 17 33 PM

Predicatble loss of Syn-packets with epcap/pkt: (Was: Unsupported packet decoding from WLAN interface)

When connecting Epcap via a Wlan interface to a DSL router I get the following message:

pkt:decode(A1).                                              
 {error,[#ether{dhost = <<131,104,5,100,0,6>>,                                                  shost = <<"packet">>,type = 24833,crc = 0}],
       {unsupported,<<104,3,98,0,0,5,108,98,0,2,125,164,98,0,0,
                      114,174,97,74,109,0,0,0,74,...>>}}

pkt:decode(A2).
{error,[#ether{dhost = <<131,104,5,100,0,6>>,
               shost = <<"packet">>,type = 24833,crc = 0}],
       {unsupported,<<104,3,98,0,0,5,108,98,0,2,125,164,98,0,0,
                      114,253,97,66,109,0,0,0,66,...>>}}

pkt:decode(A3).
{error,[#ether{dhost = <<131,104,5,100,0,6>>,
               shost = <<"packet">>,type = 24833,crc = 0}],
       {unsupported,<<104,3,98,0,0,5,108,98,0,2,125,164,98,0,0,
                      115,224,97,176,109,0,0,0,176,...>>}}

where

A1 = <<131,104, 5,100,0, 6,112, 97,99, 107,101, 116,97, 1,104,3, 98,0,0, 5,108, 98,0,2, 125,164, 98,0,0, 114,174, 97,74, 109,0,0, 0,74, 160,243, 193,39, 121,154, 116,49, 112,96, 174,251, 8,0,69, 0,0,60, 136,220, 64,0, 246,6, 231,22, 193,99, 144,85, 192,168, 2,103,0, 80,171, 194,104, 40,87, 32,132, 24,125, 153,160, 18,17,4, 29,86,0, 0,2,4,5, 172,1,1, 8,10,52, 250,1, 135,0, 100,99, 236,4,2, 0,0>>.

A2 = <<131,104, 5,100,0, 6,112, 97,99, 107,101, 116,97, 1,104,3, 98,0,0, 5,108, 98,0,2, 125,164, 98,0,0, 114,253, 97,66, 109,0,0, 0,66, 116,49, 112,96, 174,251, 160,243, 193,39, 121,154, 8,0,69, 0,0,52, 30,117, 64,0,64, 6,7,135, 192,168, 2,103, 193,99, 144,85, 171,194, 0,80, 132,24, 125,153, 104,40, 87,33, 128,16, 114,16, 231,253, 0,0,1,1, 8,10,0, 100,99, 243,52, 250,1, 135>>.

A3 = <<131,104, 5,100,0, 6,112, 97,99, 107,101, 116,97, 1,104,3, 98,0,0, 5,108, 98,0,2, 125,164, 98,0,0, 115,224, 97,176, 109,0,0, 0,176, 116,49, 112,96, 174,251, 160,243, 193,39, 121,154, 8,0,69, 0,0,162, 30,118, 64,0,64, 6,7,24, 192,168, 2,103, 193,99, 144,85, 171,194, 0,80, 132,24, 125,153, 104,40, 87,33, 128,24, 114,16, 194,51, 0,0,1,1, 8,10,0, 100,99, 243,52, 250,1, 135,71, 69,84, 32,47, 32,72, 84,84, 80,47, 49,46, 49,13, 10,85, 115,101, 114,45, 65,103, 101,110, 116,58, 32,87, 103,101, 116,47, 49,46, 49,52, 32,40, 108,105, 110,117, 120,45, 103,110, 117,41, 13,10, 65,99, 99,101, 112,116, 58,32, 42,47, 42,13, 10,72, 111,115, 116,58, 32,119, 119,119, 46,104, 101,105, 115,101, 46,100, 101,13, 10,67, 111,110, 110,101, 99,116, 105,111, 110,58, 32,75, 101,101, 112,45, 65,108, 105,118, 101,13, 10,13,10>>

and EPCAP is called with the following parameters:

[{interface,"wlan2"},{cluster_id,2},{filter,"tcp"},{group,"root"}]

makesum does not work for ipv4 packets

Makesum fails for normal ipv4 packets. Here is my code:

        [_EtherIgnore, IP, TCP, PayloadPadded] = pkt:decapsulate({pkt:link_type(DLT), Packet}),
            PayloadSize = payloadsize(IP, TCP),
            Payload = <<PayloadPadded:PayloadSize/binary>>,
        case IP of
            #ipv4{} ->
                IPSum = pkt:makesum(IP#ipv4{sum = 0}),
            TCPSum = pkt:makesum([IP, TCP#tcp{sum = 0}, Payload]),
            case [IPSum, TCPSum] of 
            [0,0] ->
                ok;
                    [_,_] ->
                io:format("Wrong checksum: {IPSum~p, TCPSum~p}~n Packet:~p~n", [IPSum, TCPSum, Packet])
                end;
            #ipv6{} ->
            ok % checksum not implemented for ipv6 
        end,

See related issue:
msantos/epcap#12

Here are a few sample packets:

Wrong checksum: {IPSum20593, TCPSum30697}
 Packet:<<0,37,34,169,124,93,156,199,166,109,119,220,8,0,69,0,3,60,65,182,64,0,
          57,6,80,113,93,184,220,29,192,168,178,22,0,80,138,123,234,253,167,67,
          56,80,67,95,80,24,1,23,119,233,0,0,72,84,84,80,47,49,46,49,32,50,48,
          48,32,79,75,13,10,65,99,99,101,112,116,45,82,97,110,103,101,115,58,
          32,98,121,116,101,115,13,10,67,97,99,104,101,45,67,111,110,116,114,
          111,108,58,32,109,97,120,45,97,103,101,61,53,49,50,54,49,55,13,10,67,
          111,110,116,101,110,116,45,84,121,112,101,58,32,97,112,112,108,105,
          99,97,116,105,111,110,47,111,99,115,112,45,114,101,115,112,111,110,
          115,101,13,10,68,97,116,101,58,32,83,117,110,44,32,50,48,32,79,99,
          116,32,50,48,49,51,32,50,49,58,50,54,58,51,56,32,71,77,84,13,10,69,
          116,97,103,58,32,34,53,50,54,52,50,49,99,53,45,49,100,55,34,13,10,69,
          120,112,105,114,101,115,58,32,83,117,110,44,32,50,55,32,79,99,116,32,
          50,48,49,51,32,48,57,58,50,54,58,51,56,32,71,77,84,13,10,76,97,115,
          116,45,77,111,100,105,102,105,101,100,58,32,83,117,110,44,32,50,48,
          32,79,99,116,32,50,48,49,51,32,49,56,58,51,50,58,51,55,32,71,77,84,
          13,10,83,101,114,118,101,114,58,32,69,67,83,32,40,102,114,97,47,68,
          52,54,70,41,13,10,88,45,67,97,99,104,101,58,32,72,73,84,13,10,67,111,
          110,116,101,110,116,45,76,101,110,103,116,104,58,32,52,55,49,13,10,
          13,10,48,130,1,211,10,1,0,160,130,1,204,48,130,1,200,6,9,43,6,1,5,5,
          7,48,1,1,4,130,1,185,48,130,1,181,48,129,158,162,22,4,20,80,234,115,
          137,219,41,251,16,143,158,229,1,32,212,222,121,153,72,131,247,24,15,
          50,48,49,51,49,48,50,48,49,56,49,55,51,53,90,48,115,48,113,48,73,48,
          9,6,5,43,14,3,2,26,5,0,4,20,237,72,173,221,203,123,0,226,14,132,42,
          169,180,9,241,172,48,52,207,150,4,20,80,234,115,137,219,41,251,16,
          143,158,229,1,32,212,222,121,153,72,131,247,2,16,9,206,42,242,11,72,
          43,120,162,22,63,155,169,189,115,246,128,0,24,15,50,48,49,51,49,48,
          50,48,49,56,49,55,51,53,90,160,17,24,15,50,48,49,51,49,48,50,55,49,
          56,51,50,51,53,90,48,13,6,9,42,134,72,134,247,13,1,1,5,5,0,3,130,1,1,
          0,128,119,229,248,128,61,221,85,232,34,45,24,246,204,176,159,9,242,
          72,143,11,65,118,219,127,219,78,32,176,67,114,144,221,242,63,231,113,
          111,209,112,175,81,10,187,48,138,112,128,177,140,232,46,80,158,21,67,
          230,37,160,96,191,201,176,152,223,114,173,97,239,212,249,69,20,220,
          217,146,106,178,63,153,25,183,203,7,173,122,49,46,134,128,10,107,187,
          66,226,111,90,9,216,14,231,85,140,99,198,202,236,207,212,199,150,89,
          129,44,58,137,176,181,156,57,230,75,161,5,30,130,196,188,137,32,240,
          162,54,244,229,173,216,133,154,234,56,39,107,111,184,153,33,45,165,
          43,141,218,190,88,45,83,202,29,47,54,1,185,240,35,152,35,254,187,181,
          105,229,180,186,147,88,83,117,90,216,187,53,27,65,94,66,203,154,210,
          140,110,121,112,105,151,171,55,68,229,3,145,177,44,57,203,163,5,157,
          150,168,25,129,84,216,149,71,114,78,140,107,201,156,18,186,55,210,73,
          124,194,54,234,185,237,63,199,110,99,162,130,122,179,206,82,185,239,
          68,28,92,246,244,203,21,154,150,152,41,29>>
Wrong checksum: {IPSum10272, TCPSum26328}
 Packet:<<156,199,166,109,119,220,0,37,34,169,124,93,8,0,69,0,0,40,102,27,64,0,
          64,6,40,32,192,168,178,22,93,184,220,29,138,123,0,80,56,80,67,95,234,
          253,170,87,80,16,0,151,102,216,0,0>>
Wrong checksum: {IPSum21380, TCPSum26199}
 Packet:<<0,37,34,169,124,93,156,199,166,109,119,220,8,0,69,0,0,40,65,183,64,0,
          57,6,83,132,93,184,220,29,192,168,178,22,0,80,138,123,234,253,170,87,
          56,80,67,95,80,17,1,23,102,87,0,0,0,0,0,0,0,0>>
Wrong checksum: {IPSum10271, TCPSum26326}
 Packet:<<156,199,166,109,119,220,0,37,34,169,124,93,8,0,69,0,0,40,102,28,64,0,
          64,6,40,31,192,168,178,22,93,184,220,29,138,123,0,80,56,80,67,95,234,
          253,170,88,80,17,0,151,102,214,0,0>>
Wrong checksum: {IPSum21379, TCPSum26198}
 Packet:<<0,37,34,169,124,93,156,199,166,109,119,220,8,0,69,0,0,40,65,184,64,0,
          57,6,83,131,93,184,220,29,192,168,178,22,0,80,138,123,234,253,170,88,
          56,80,67,96,80,16,1,23,102,86,0,0,0,0,0,0,0,0>>

Checksum calculation is slow

Here is a eprof log when running josemic/enose (defrag branch) using msantos/epcap and msantos/pkt while downloading a file at 1.3 MByte/s.

Offloading is switched off.

Most of the time is spent in the functions pkt:comp and pkt:checksum.
Any ideas on how to speed this up?

28> eprof:analyze(total).                                            
FUNCTION                                                                  CALLS      %     TIME  [uS / CALLS]
--------                                                                  -----    ---     ----  [----------]
io:o_request/3                                                                1   0.00        2  [      2.00]
io:format/3                                                                   1   0.00        3  [      3.00]
io:execute_request/2                                                          1   0.00        3  [      3.00]
erlang:group_leader/0                                                         1   0.00        3  [      3.00]
io:bc_req/3                                                                   1   0.00        4  [      4.00]
gen:do_call/4                                                                 1   0.00        4  [      4.00]
gen_server:call/3                                                             1   0.00        4  [      4.00]
supervisor:call/2                                                             1   0.00        4  [      4.00]
net_kernel:dflag_unicode_io/1                                                 1   0.00        4  [      4.00]
erlang:whereis/1                                                              1   0.00        5  [      5.00]
erlang:make_ref/0                                                             1   0.00        5  [      5.00]
io:request/2                                                                  1   0.00        6  [      6.00]
io:default_output/0                                                           1   0.00        6  [      6.00]
defrag_server:insert_element/2                                                1   0.00        6  [      6.00]
supervisor:start_child/2                                                      1   0.00        6  [      6.00]
io:format/2                                                                   1   0.00        7  [      7.00]
io:io_request/2                                                               1   0.00        7  [      7.00]
gen:call/4                                                                    1   0.00        7  [      7.00]
defrag_root_sup:start_worker/3                                                1   0.00        9  [      9.00]
io:wait_io_mon_reply/2                                                        1   0.00       10  [     10.00]
erlang:setelement/3                                                           2   0.00       11  [      5.50]
erlang:demonitor/2                                                            2   0.00       14  [      7.00]
erlang:ref_to_list/1                                                          1   0.00       16  [     16.00]
erlang:list_to_atom/1                                                         1   0.00       16  [     16.00]
erlang:monitor/2                                                              2   0.00       17  [      8.50]
erlang:send/3                                                                 1   0.00       20  [     20.00]
parser_combinator_bitstring:binary_revers/1                                   4   0.00       22  [      5.50]
parser_combinator_bitstring:binary_revers/2                                  10   0.00       28  [      2.80]
parser_combinator_bitstring:matchCaseInsensitive/3                           30   0.00       94  [      3.13]
parser_combinator_bitstring:'-pBinarystringCaseInsensitive/1-fun-0-'/2       30   0.00      162  [      5.40]
defrag_server:handle_info/2                                                3250   0.03    10841  [      3.34]
gen_server:handle_common_reply/6                                           3249   0.03    10978  [      3.38]
s6:'-s/0-fun-0-'/1                                                         2042   0.04    12754  [      6.25]
s6:'-s/0-fun-1-'/1                                                         2042   0.04    12965  [      6.35]
parser_combinator_bitstring:'-pThen/2-fun-0-'/3                            4084   0.04    13434  [      3.29]
pkt_ether:type/1                                                           6500   0.04    14914  [      2.29]
parser_combinator_bitstring:'-pBetweenN/3-fun-0-'/4                        4084   0.04    15617  [      3.82]
pkt_tcp:options/1                                                          3250   0.04    15959  [      4.91]
pkt:ipproto/1                                                              6500   0.04    16307  [      2.51]
pkt_dlt:codec/1                                                            6500   0.05    16823  [      2.59]
pkt_ipproto:codec/1                                                        6500   0.05    17206  [      2.65]
defrag_worker:send_packet/2                                                3248   0.05    18095  [      5.57]
pkt:ether_type/1                                                           6500   0.05    18949  [      2.92]
epcap_port_lib:header/1                                                    3250   0.05    19190  [      5.90]
epcap_port_lib:tcp_flags/1                                                 3250   0.05    19954  [      6.14]
epcap_port_lib:decode/2                                                    3250   0.06    22116  [      6.80]
defrag_server:get_connection_worker_pid_by_address_tuple/2                 3254   0.06    22749  [      6.99]
pkt_ether:codec/1                                                          6500   0.06    23333  [      3.59]
parser_combinator_bitstring:pBetweenN/3                                    4084   0.07    24725  [      6.05]
gen_server:handle_msg/5                                                    7334   0.07    25430  [      3.47]
gen_server:decode_msg/8                                                    7334   0.07    25797  [      3.52]
lists:reverse/2                                                            9747   0.07    26658  [      2.73]
parser_combinator_bitstring:pThen/2                                        4084   0.08    28802  [      7.05]
parser_combinator_bitstring:pBinarystringCaseInsensitive/1                 4084   0.08    29844  [      7.31]
lists:reverse/1                                                            9750   0.08    30326  [      3.11]
pkt:dlt/1                                                                  6500   0.08    30687  [      4.72]
lists:member/2                                                            13000   0.09    31471  [      2.42]
parser_combinator_bitstring:pBinarystring/1                                4084   0.09    32389  [      7.93]
defrag_server:payloadsize/2                                                3250   0.09    32952  [     10.14]
content_worker:handle_call/3                                               4084   0.09    34026  [      8.33]
pkt_tcp:options/2                                                         12995   0.10    34730  [      2.67]
pkt:decapsulate/1                                                          6500   0.10    35189  [      5.41]
inet_parse:ntoa/1                                                          6500   0.10    35428  [      5.45]
pkt:ether/1                                                                6500   0.10    36081  [      5.55]
parser_combinator_bitstring:'-pUntilN/2-fun-0-'/3                          4086   0.10    36959  [      9.05]
epcap_port_lib:port/2                                                      6500   0.10    36995  [      5.69]
pkt_tcp:codec/1                                                            9750   0.11    38266  [      3.92]
parser_combinator_bitstring:parse/2                                        8168   0.11    40590  [      4.97]
pkt:makesum/1                                                              6500   0.11    40611  [      6.25]
lists:keyfind/3                                                           16250   0.12    42029  [      2.59]
pkt:ipv4/1                                                                 9750   0.14    49418  [      5.07]
parser_combinator_bitstring:pUntilN/2                                      8168   0.14    51463  [      6.30]
gen_server:reply/2                                                         4084   0.14    52419  [     12.84]
pkt:checksum/1                                                            13000   0.14    52540  [      4.04]
pkt_ipv4:codec/1                                                           9750   0.15    53636  [      5.50]
erlang:'++'/2                                                             19502   0.16    57459  [      2.95]
pkt:tcp/1                                                                  9750   0.18    64046  [      6.57]
pkt:decapsulate_next/2                                                    26000   0.18    64203  [      2.47]
pkt:next/1                                                                13000   0.19    69929  [      5.38]
gen_server:loop/6                                                          7333   0.20    73841  [     10.07]
epcap_port_lib:'-tcp_flags/1-lc$^0/1-0-'/1                                29250   0.23    83148  [      2.84]
erlang:integer_to_list/1                                                  26001   0.24    88303  [      3.40]
gen_server:dispatch/3                                                      3250   0.28   101920  [     31.36]
gen_fsm:send_event/2                                                       3248   2.38   865274  [    266.40]
parser_combinator_bitstring:untilN/4                                     412114   3.94  1432868  [      3.48]
parser_combinator_bitstring:match/3                                      413700   4.41  1602864  [      3.87]
parser_combinator_bitstring:'-pBinarystring/1-fun-0-'/2                  412084   8.17  2974405  [      7.22]
pkt:compl/2                                                             1635928  10.43  3793934  [      2.32]
lists:foldl/3                                                           1642427  10.49  3816434  [      2.32]
pkt:compl/1                                                             1635927  10.53  3830908  [      2.34]
pkt:'-checksum/1-fun-0-'/2                                              1635928  20.27  7376856  [      4.51]
pkt:'-checksum/1-blc$^0/1-0-'/1                                         1642936  24.18  8798810  [      5.36]

Enhancement request

Currently, there is an API pkt:decode(Data) to decode binary to various records provided by pkt library. It would be good to have an API for pkt:encode(Data) as well to convert records in pkt to binary data. This way, a caller can fill the required records and have the list encoded to a binary as a single API. Also, the encode function would give symmetricity to the APIs you have provided.

IPv4 Checksum fails (due to tcp offloading)

I have the case that wireshark and pkt calculate (the identical) checksum error, but my linux pc acknowledges the packet.
Thus either wireshark and pkt have both the same error, pcap has an error or my Linux-PC does not calculate the TCP checksum before acknowledging the packet.

screenshot from 2013-10-23 17 45 53

Fame = <<0,37,34,169,124,93,156,199,166,109,119,220,8,0,69,0,5,220,122,52,64,
          0,247,6,63,111,193,99,144,85,192,168,178,22,0,80,198,169,201,148,4,
          83,16,103,129,8,128,24,17,80,202,70,0,0,1,1,8,10,229,143,96,87,0,21,
          41,232,72,84,84,80,47,49,46,49,32,50,48,48,32,79,75,13,10,68,97,116,
          101,58,32,87,101,100,44,32,50,51,32,79,99,116,32,50,48,49,51,32,49,
          51,58,52,51,58,53,48,32,71,77,84,13,10,83,101,114,118,101,114,58,32,
          65,112,97,99,104,101,13,10,69,120,112,105,114,101,115,58,32,87,101,
          100,44,32,50,51,32,79,99,116,32,50,48,49,51,32,49,51,58,52,56,58,53,
          49,32,71,77,84,13,10,86,97,114,121,58,32,85,115,101,114,45,65,103,
          101,110,116,44,65,99,99,101,112,116,45,69,110,99,111,100,105,110,103,
          13,10,67,97,99,104,101,45,99,111,110,116,114,111,108,58,32,109,97,
          120,45,97,103,101,61,51,48,48,13,10,88,45,67,111,98,98,108,101,114,
          58,32,111,99,116,111,49,52,46,104,101,105,115,101,46,100,101,13,10,
          67,111,110,110,101,99,116,105,111,110,58,32,99,108,111,115,101,13,10,
          84,114,97,110,115,102,101,114,45,69,110,99,111,100,105,110,103,58,32,
          99,104,117,110,107,101,100,13,10,67,111,110,116,101,110,116,45,84,
          121,112,101,58,32,116,101,120,116,47,104,116,109,108,59,32,99,104,97,
          114,115,101,116,61,117,116,102,45,56,13,10,13,10,49,97,48,98,54,13,
          10,60,33,68,79,67,84,89,80,69,32,104,116,109,108,62,10,60,104,116,
          109,108,32,108,97,110,103,61,34,100,101,34,62,10,10,60,104,101,97,
          100,62,10,10,32,32,32,32,60,116,105,116,108,101,62,73,84,45,78,101,
          119,115,44,32,99,39,116,44,32,105,88,44,32,84,101,99,104,110,111,108,
          111,103,121,32,82,101,118,105,101,119,44,32,84,101,108,101,112,111,
          108,105,115,32,124,32,104,101,105,115,101,32,111,110,108,105,110,101,
          60,47,116,105,116,108,101,62,10,32,32,32,32,32,32,32,32,60,109,101,
          116,97,32,110,97,109,101,61,34,100,101,115,99,114,105,112,116,105,
          111,110,34,32,99,111,110,116,101,110,116,61,34,78,101,119,115,32,117,
          110,100,32,70,111,114,101,110,32,122,117,32,67,111,109,112,117,116,
          101,114,44,32,73,84,44,32,87,105,115,115,101,110,115,99,104,97,102,
          116,44,32,77,101,100,105,101,110,32,117,110,100,32,80,111,108,105,
          116,105,107,46,32,80,114,101,105,115,118,101,114,103,108,101,105,99,
          104,32,118,111,110,32,72,97,114,100,119,97,114,101,32,117,110,100,32,
          83,111,102,116,119,97,114,101,32,115,111,119,105,101,32,68,111,119,
          110,108,111,97,100,115,32,98,101,105,109,32,72,101,105,115,101,32,90,
          101,105,116,115,99,104,114,105,102,116,101,110,32,86,101,114,108,97,
          103,46,34,32,47,62,10,32,32,32,32,32,32,32,32,32,32,32,32,60,109,101,
          116,97,32,110,97,109,101,61,34,107,101,121,119,111,114,100,115,34,32,
          99,111,110,116,101,110,116,61,34,104,101,105,115,101,32,111,110,108,
          105,110,101,44,32,99,39,116,44,32,105,88,44,32,84,101,99,104,110,111,
          108,111,103,121,32,82,101,118,105,101,119,44,32,78,101,119,115,116,
          105,99,107,101,114,44,32,84,101,108,101,112,111,108,105,115,44,32,83,
          101,99,117,114,105,116,121,44,32,78,101,116,122,101,34,32,47,62,10,
          10,32,32,32,32,10,10,10,10,60,109,101,116,97,32,99,104,97,114,115,
          101,116,61,34,117,116,102,45,56,34,62,10,60,109,101,116,97,32,110,97,
          109,101,61,34,112,117,98,108,105,115,104,101,114,34,32,99,111,110,
          116,101,110,116,61,34,72,101,105,115,101,32,90,101,105,116,115,99,
          104,114,105,102,116,101,110,32,86,101,114,108,97,103,34,32,47,62,10,
          60,109,101,116,97,32,110,97,109,101,61,34,118,105,101,119,112,111,
          114,116,34,32,99,111,110,116,101,110,116,61,34,119,105,100,116,104,
          61,100,101,118,105,99,101,45,119,105,100,116,104,44,32,105,110,105,
          116,105,97,108,45,115,99,97,108,101,61,49,46,48,34,32,47,62,60,108,
          105,110,107,32,114,101,108,61,34,104,111,109,101,34,32,116,121,112,
          101,61,34,116,101,120,116,47,104,116,109,108,34,32,116,105,116,108,
          101,61,34,83,116,97,114,116,115,101,105,116,101,34,32,104,114,101,
          102,61,34,47,34,32,47,62,10,60,108,105,110,107,32,114,101,108,61,34,
          99,111,112,121,114,105,103,104,116,34,32,116,105,116,108,101,61,34,
          67,111,112,121,114,105,103,104,116,34,32,104,114,101,102,61,34,47,73,
          109,112,114,101,115,115,117,109,45,52,56,54,50,46,104,116,109,108,34,
          32,47,62,32,32,32,32,32,10,32,32,32,32,10,32,32,32,32,32,32,32,32,60,
          33,45,45,103,111,111,103,108,101,111,102,102,58,32,97,108,108,45,45,
          62,10,32,32,32,32,32,32,32,32,60,109,101,116,97,32,110,97,109,101,61,
          34,116,119,105,116,116,101,114,58,99,97,114,100,34,32,32,32,99,111,
          110,116,101,110,116,61,34,115,117,109,109,97,114,121,34,32,47,62,10,
          32,32,32,32,32,32,32,32,60,109,101,116,97,32,110,97,109,101,61,34,
          116,119,105,116,116,101,114,58,115,105,116,101,34,32,32,32,99,111,
          110,116,101,110,116,61,34,64,104,101,105,115,101,111,110,108,105,110,
          101,34,32,47,62,10,32,32,32,32,32,32,32,32,60,109,101,116,97,32,110,
          97,109,101,61,34,116,119,105,116,116,101,114,58,100,111,109,97,105,
          110,34,32,99,111,110,116,101,110,116,61,34,104,101,105,115,101,46,
          100,101,34,32,47,62,10,32,32,32,32,60,109,101,116,97,32,112,114,111,
          112,101,114,116,121,61,34,102,98,58,112,97,103,101,95,105,100,34,32,
          32,32,32,32,99,111,110,116,101,110,116,61,34,51,51,51,57,57,50,51,54,
          55,51,49,55,34,32,47,62,10,32,32,32,32,60,109,101,116,97,32,112,114,
          111,112,101,114,116,121,61,34,111,103,58,116,105,116,108,101,34,32,
          32,32,32,32,32,32,99,111,110,116,101,110,116,61,34,73,84,45,78,101,
          119,115,44,32,99,38,35,51,57,59,116,44,32,105,88,44,32,84,101,99,104,
          110,111,108,111,103,121,32,82,101,118,105,101,119,44,32,84,101,108,
          101,112,111,108,105,115,34,32,47,62,10,32,32,32,32,60,109,101,116,97,
          32,112,114,111,112,101,114,116>>

[#ether{}, #ipv4{sum = IPSum} = IP, #tcp{sum = TCPSum} = TCP, Payload] = pkt:decapsulate(Frame).

[#ether{dhost = <<0,37,34,169,124,93>>,
        shost = <<156,199,166,109,119,220>>,
        type = 2048,crc = 0},
 #ipv4{v = 4,hl = 5,tos = 0,len = 1500,id = 31284,df = 1,
       mf = 0,off = 0,ttl = 247,p = 6,sum = 16239,
       saddr = {193,99,144,85},
       daddr = {192,168,178,22},
       opt = <<>>},
 #tcp{sport = 80,dport = 50857,seqno = 3381920851,
      ackno = 275218696,off = 8,cwr = 0,ece = 0,urg = 0,ack = 1,
      psh = 1,rst = 0,syn = 0,fin = 0,win = 4432,sum = 51782,
      urp = 0,
      opt = <<1,1,8,10,229,143,96,87,...>>},
 <<"HTTP/1.1 200 OK\r\nDate: Wed, 23 Oct 2013 13:43:50 GMT\r\nServer: Apache\r\nExpires: Wed, 23 Oct 2013 13:4"...>>]

> pkt:makesum([IP, TCP#tcp{sum = 0}, Payload]).
26846
> IPSum = pkt:makesum(IP#ipv4{sum = 0}).                    
16239

The pcap file is available upon request. Any ideas?

Failure in decoding ARP / ARP reply

I want to receive packets and check if they are ARP replies.
Thus, I look for 'arp' in the Frame, but it never matches. I couldn't find a single ARP packet which was decoded.


receive
        {Port, {data, Data}} -> 
            Frame = pkt:decapsulate(Data),
            ARP = lists:keyfind(arp,1,Frame),
            io:format("Frame: ~p~n", [Frame]),
            io:format("Data: ~p~n", [Data]),
            case ARP of
                false ->
                    ok;
                _ ->
                    Type = ARP#arp.op,
                    case Type of
                        ?ARPOP_REPLY ->
                            % Get the ARP sender ip which is the key
                            Key = ARP#arp.sip,
                            (#{Key := ListData} = Queue),
                            DestMAC = ARP#arp.sha,
                            io:format("ARP_REPLY Key: ~p~n", [Key]),
                            io:format("ARP_REPLY DestMac: ~p~n", [DestMAC]),
                            sendQueuedFrames(ListData, DestMAC, ListenSock, IfIndex);
                        _ ->
                            ok
                    end
            end
    end

Am I wrong or is there a bug on decoding?
Thanks!

Detection of the IEEE 802.3 ethernet type and LLC codec

Please review the dev branch, it's adds detection of the IEEE 802.3 ethernet type and Logical-Link Control codec.

Detection is based on the length value of the ethernet frame, in case of IEEE 802.3 ethernet it's should less than 1500.

@msantos, why do we need the ETH_P_ALL in the include/pkt_ether.hrl?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.