rsmmr / hilti Goto Github PK

Setting the bitorder globally for the module apparently has no effect:

# cat mini.spicy
module  Mini;

%bitorder=Spicy::BitOrder::MSB0;

export type Test = unit {
    bits : bitfield(8) {
        b0: 0;
        middle: 1..6;
        b7: 7;
        };

    on %done {
        print self;
    }
};
# printf "\x01" | spicy-driver mini.spicy
<bits=(b0=1, middle=0, b7=0)>

Expected output: <bits=(b0=0, middle=0, b7=1)>.
Compare this with setting the bitorder for the field, which produces the desired result:

# cat mini-field.spicy
module  Mini;

export type Test = unit {
    bits : bitfield(8) {
        b0: 0;
        middle: 1..6;
        b7: 7;
        } &bitorder=Spicy::BitOrder::MSB0;

    on %done {
        print self;
    }
};
# printf "\x01" | spicy-driver mini-field.spicy
<bits=(b0=0, middle=0, b7=1)>

The hilti/tuple/wrong-index unit test fails because of a segmentation fault:

# btest -d hilti/tuple/wrong-index.hlt
[ 33%] hilti.tuple.wrong-index-2 ... failed
  % 'btest-diff output' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===============================

  >>> b = tuple.index a b
  wrong-index.hlt:8-9: error, operand must be a constant [pass::hilti::Validator]
  Segmentation fault
  error running hiltic, aborting
  == Diff ===============================
  --- /tmp/test-diff.2274.output.baseline.tmp   2017-01-13 10:27:06.998718128 +0000
  +++ /tmp/test-diff.2274.output.tmp    2017-01-13 10:27:07.038688438 +0000
  @@ -1,4 +1,5 @@

  >>> b = tuple.index a b
  wrong-index.hlt:XXX: error, operand must be a constant [pass::hilti::Validator]
  +Segmentation fault
  error running hiltic, aborting
  =======================================

  % cat .stderr

1 of 3 tests failed

Tested in the rsmmr/hilti Docker image:

# hilti-config --version
HILTI 0.5-4

Declaring a function after the unit that it's used in results in a compilation error:

# cat mini.spicy
module Mini;

export type MyUnit = unit {
    field : bytes &convert=myfunc($$);
};

bytes myfunc(b: bytes) {
    return b;
}

# hilti-build -o mini mini.spicy

>>> ref<bytes> myfunc__Vjw(ref<bytes> b, <Unresolved 'SpicyHilti::UserCookie'> __cookie) {
  return.result b  }
mini.hb689.tmp.hlt:532-532: error, ID myfunc__Vjw already declared [pass::hilti::ScopeBuilder]
mini.hb689.tmp.hlt: Aborting due to verification error.
error running hiltic, aborting

Declaring the function before the unit results in a successful compilation:

# cat mini-reordered.spicy
module Mini;

bytes myfunc(b: bytes) {
    return b;
}

export type MyUnit = unit {
    field : bytes &convert=myfunc($$);
};

# hilti-build -o mini-reordered mini-reordered.spicy
#  

Hello,
I think there is an issue to store in a variable the value pointed by an iter<bytes>.

import BinPAC;

global b : bytes = b"Hello";
global it : iter<bytes>;
global v : bytes;

it = b.begin();
print *it;
v = *it;

Raise the following error :

>>> v = assign __tmp_elem
<no location>:: error, operand not compatible with target [pass::hilti::Validator]

If we look at the hilti generated code:

void __entry()
{
  local iterator<bytes> __tmp_iter
  local int<8> &hostapp_type=(37) __tmp_elem
  local int<8> &hostapp_type=(37) __tmp_elem_2


    # Statement: it  = b .begin()
    __tmp_iter = begin b
    it = assign __tmp_iter
    # Statement: v  = *it
    __tmp_elem = deref it
   = assign __tmp_elem
    # Statement: print *it ;
    __tmp_elem_2 = deref it
    call BinPACHilti::print (__tmp_elem_2)
    block.end
}

[...]

global iterator<bytes> it
global ref<bytes> v

In fact deref i returns an int<8> for iter<bytes> however v is a ref<bytes>.
It seems it is not possible in BinPAC++ to save the value in a int<8> because the type does not match with the type of the iterator (bytes)

A solution could be to change the return type of iterator::Deref operator in the binpac compiler and use int<8> only for bytes ?

This segfaults (towards termination, I believe):

bro -r /opt/hilti/bro/test/Traces/dns.trace /opt/hilti/bro/pac2/http.evt

# cat mini.spicy
module Mini;

export type Test = unit {
    b : uint8;
};
# printf "\xFE" | spicy-dump mini.spicy
<
  b=-2
>

Correct would be b=254 (which is what spicy-driver shows if a "print self;" hook is added).

# spicy-config --version
Spicy 0.5-4

It would be useful to have more possibilities to compare enum labels. Currently, only a test for equality is supported (==). > and < would be useful for some cases, for example the following, where a range of numbers defined in an enum needs to be queried:

type ExampleEnum = enum {
  # ...
  Foo1 = 0xFEFC,
  Foo2 = 0xFEFD,
  Foo3 = 0xFEFE,
  Foo4 = 0xFEFF,
  # ...
};

type ExampleUnit = unit() {
  frame_id: uint16 &convert=ExampleEnum($$);
  ... if (self.frame_id >= ExampleEnum::Foo1 && self.frame_id <= ExampleEnum::Foo2);
}

It would be nice if this would work for not converted integers, as well:

  frame_id: uint16;

Hello !
It seems that "&convert" in a bitfield is not well supported.

type Foo = enum{BAR1=0, BAR2=1};

export type test = unit{
    f1 : bitfield(8){
            b1 : 0..3 &convert(Foo($$));
            b2 : 4..7;
    }; 
    on f1{
        print self.f1.b1;
        if (self.f1.b1 == Foo::BAR1){
            print "f1.b1 BAR1";
        }
    }
};

Through an error because of the if between self.f1.b1 and Foo::BAR1

target: bool
op1 : Foo
op2: int<64>
op3 : (null)
internal error, llvmInstruction: 0 matches for mnemo equal [codegen]

I also noticed that $$ type is bitfield

However the print self.f1.b1 shows the good value. (Corresponds to the unit test https://github.com/rsmmr/hilti/blob/master/tests/binpac/enum/parse-bitfield-enum.pac2 )

# btest -d spicy/double/parse.spicy
[  0%] spicy.double.parse ... failed
  % 'btest-diff output' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===============================
  <f1=2814768777216304605379219959026329549267817057258717999296725371294903577351399237811009219760593733213166659574875750222176724, f2=2821128616056011089531440770821113512346859011793081747591869489872827629227737079711884337401482318061052193593026198347157899, f3=279523120233054208.000000>
  == Diff ===============================
  --- /tmp/test-diff.2422.output.baseline.tmp   2017-01-13 10:32:11.422642241 +0000
  +++ /tmp/test-diff.2422.output.tmp    2017-01-13 10:32:11.462612547 +0000
  @@ -1 +1 @@
  -<f1=12.345679, f2=12.345679, f3=3.140000>
  +<f1=2814768777216304605379219959026329549267817057258717999296725371294903577351399237811009219760593733213166659574875750222176724, f2=2821128616056011089531440770821113512346859011793081747591869489872827629227737079711884337401482318061052193593026198347157899, f3=279523120233054208.000000>
  =======================================

  % cat .stderr

1 of 1 test failed

Tested in the rsmmr/hilti Docker image:

# spicy-config --version
Spicy 0.5-4
# hilti-config --version
HILTI 0.5-4

HILTI and BinPAC's UTF8 parsing gets confused with a sequence of 54 65 6d 70 65 72 61 74 75 75 72 20 31 30 4b 20 2d 34 30 2d 31 35 30 b0 43 20 41 49 43

I am developing a protocol analyzer using Spicy, using the BRO/HILTI/SPICY docker image posted at the following URL: https://hub.docker.com/r/rsmmr/hilti/

In my Spicy protocol analyzer, I am trying to match a pattern. My data type is ‘bytes’, and I am using the ‘match()’ method. If the pattern includes an extended ASCII character (range from 0x80 to 0xFF), then the pattern fails to find a match. However, if I wildcard the extended ASCCI character, then it finds a match. While I cannot share source code from my original project, I created a sample project to demonstrate the bug.

I will upload the following sample files so that you may attempt to reproduce the bug:
(1) regex_test.spicy*
(2) regex_test.evt*
(3) regex_test.bro*
(4) smb-browser-elections.pcap

NOTE *: There might be CR/LF issues because I saved these files with Notepad on a Windows box. Also, I had to give each file a *.txt extension, in order for the file uploader to accept it.

As my sample data, I downloaded an SMB pcap file from wireshark.org. The regular expression patterns below are based on Frame #3, NetBIOS/SMB datagram, in the SMB pcap file 'smb-browser-elections.pcap' downloaded from the wireshark website, at the following URL:
https://wiki.wireshark.org/SampleCaptures?action=AttachFile&do=get&target=smb-browser-elections.pcapng

Here are my sample regex patterns**:

//# Appears at offset 0x2A in Frame 3
//# or offset 0x00 within UDP payload
const SmbRegEx_1a = /^\x11\x02.\x16/;
const SmbRegEx_1b = /^\x11\x02\x82\x16/;

//# Appears at offset 0x2D in Frame 3
//# or offset 0x03 within UDP payload
const SmbRegEx_2a = /\x16..\x7B/;
const SmbRegEx_2b = /\x16\xC0\xA8\x7B/;

Patterns _1a and _2a match successfully, because they include the wildcard in place of the extended ASCII character(s).

Patterns _1b and _2b do not match, because they contain offending character(s) in the extended ASCII range.

NOTE **: my Spicy source code contains a third regex pattern, shown below:

//# Appears at offset 0x78 in Frame 3
//# or offset 0x4E within UDP payload
const SmbRegEx_3a = /\x43\x41\x42\x00.\x53\x4D\x42/;
const SmbRegEx_3b = /\x43\x41\x42\x00\xFF\x53\x4D\x42/;

Interestingly, this pattern fails to match for both _3a and _3b. I would expect _3a to match because it contains the wildcard. Not sure what is going wrong with this pattern. Is there a certain depth/limit at which the match() method will stop searching?

Cheers!
Mark

regex_test.bro.txt
regex_test.evt.txt
regex_test.spicy.txt
smb-browser-elections.pcap.txt

I wrote a parser for the IEC60870-5-104 protocol that utilizes lists as a wrapper just like the http parser. When I run this on captures it will crash at times. The most common crash is just before the bro_done event. A stacktrace from gdb can be found here: http://pastebin.com/z0njtRHJ

module Mini;

export type AItem = unit {
    a : uint16;
    a_list : list<BItem(self)> &count=self.a;
};

type BItem = unit(f: AItem) {
    b : uint16;
    b_list : list<CItem> &count=self.b;
};

type CItem = unit(f: AItem) {
};

# spicyc mini.spicy
Segmentation fault

Note that CItem expects an argument, but it isn't given.
Removing that parameter lets the compilation succeed:

type CItem = unit {
};

# spicyc mini.spicy
#

Tested with the rsmmr/hilti Docker image:

# spicy-config --version
Spicy 0.5-4

Extending the case presented in #10, it would be nice to have the possibility to test for a range in a switch statement.

In my case I have a uint16 with a meaning similar to an EtherType or an IP protocol number. The difference is that there are ranges of values with a similar meaning. For example, if this uint16 has a value from 0xFEFC to 0xFEFF, I want to call one particular unit for further processing, regardless of the exact value.

type ExampleEnum = enum {
  # ...
  Foo1 = 0xFEFC,
  Foo2 = 0xFEFD,
  Foo3 = 0xFEFE,
  Foo4 = 0xFEFF,
  # ...
};

type ExampleUnit = unit() {
  frame_id: uint16 &convert=ExampleEnum($$);

  switch ( self.frame_id ) {
    ExampleEnum::Foo1..ExampleEnum::Foo2 -> : OtherUnit(len - self.offset());
    # ...
    * -> : bytes &length=(len - self.offset());
  };
}

A syntax for defining the range had to be invented, one could chose two dots .. like in the definition of ranges in bitfields. This should not be restricted to enum labels but could work with all types for which > and < are defined.

Hello,

I think I have found an "use after free" bug in the map expiration timers.
I am currently implementing timeout on Spicy containers https://github.com/FrozenCaribou/hilti/tree/spicy_container_timeout

Here my pac2 example :

module Test;

import BinPAC;
import Bro;

global m: map<bytes, bytes >;

m.timeout(BinPAC::ExpireStrategy::Create, interval(60.0));

export type Foo = unit{
    t : bytes &length=1;
    key : bytes &length=4;
    value : bytes &eod;

    on %done{
        if (self.t == b"a"){
            m[self.key] = self.value;
        }
        print m;
    }
};

The evt file :

grammar ./test.pac2;

protocol analyzer pac2::Test over UDP:
    parse with Test::Foo,
    port 1337/udp;

The equivalent Hilti code of the problem:

    m = new map<string, string>
    map.timeout m Hilti::ExpireStrategy::Create interval(5.0)
    map.insert m "A-0" "test"
    timer_mgr.advance time(6.0) t

Note that map.timeout uses the context global time mgr.

When map.insert is executed, we execute this following code:
https://github.com/rsmmr/hilti/blob/master/libhilti/map_set.c#L418-L422

 __hlt_map_timer_cookie cookie = { m, keytmp };
kh_value(m, i).timer = __hlt_timer_new_map(cookie, excpt, ctx);
hlt_time t = hlt_timer_mgr_current(m->tmgr, excpt, ctx) + m->timeout;
hlt_timer_mgr_schedule(m->tmgr, t, kh_value(m, i).timer, excpt, ctx);
GC_DTOR(kh_value(m, i).timer, hlt_timer, ctx); // Not memory-managed on our end.

On map entry insertion, a timer is created and its ref_cnt = 1 (set by hlt_timer_mgr_schedule) but after the GC_DTOR the timer is unreferenced and ref_cnt = 0.
There is no visible impact while the memory is not flushed ... ( with __hlt_memory_nullbuffer_flush )
In the expire unit test ( https://github.com/rsmmr/hilti/blob/master/tests/hilti/map/expire.hlt ) we cannot see this issue because there is no flush of the nullbuffer buffer. Flush are done with safepoint which are used in Spicy parsers.
If a safepoint is created, the __hlt_memory_nullbuffer_flush function is executed, the previous address of the timer will be free (because its ref=0) and the value in the ram memory can change with other malloc/calloc executed by the application.

Then the timer manager checks timers and fires them, __hlt_timer_fire ( https://github.com/rsmmr/hilti/blob/master/libhilti/timer.c#L67 ) deals with a bad timers (data overwritten by others malloc) and produces unpredictable behavior as a segmentation fault.

I created a patch and removed in the hlt_map_insert (and for set/list/vector) the following line:

GC_DTOR(kh_value(m, i).timer, hlt_timer, ctx);  // Not memory-managed on our end.

I think it is ok because timers are also deleted when it expires, cancels or is deleted (dtor) but I am not totally sure. Here the commit : FrozenCaribou@a6f7434

This works well but there is another bug on hlt_execution_context_delete that appears when I stop Bro with Ctrl + C (Unix signal 2) whereas there are still expiration pending timers, it is during the hlt_execution_context_delete (__hlt_memory_nullbuffer_delete)
I have a bad access memory in priority_queue_remove because *mgr->timers->d is null (crashes at when getpri(q->d[posn] = 0x0 ))
I fixed the bug by checking this value before callingpriority_queue_remove. See commit : FrozenCaribou@f1ba048

I suspect that that global timer manager is destroyed before the map and so priority_queue_remove is executed after priority_queue_free which performs a hlt_free on all elements.

I do not have a good enough vision of memory management in Hilti, but from comments : timers are "not memory-managed on our end." in container timeout. There seems to be manage directly in pqueue (there are removed on priority_queue_free) so I am wondering why there are present and referenced in the nullbuffer ?
I found the commit relative to this memory management : 7b407cf

I also notice some obj ref can underflow, I will take a look on that.

I tried to import "parses/asn1.pac" but it did not work.
It seems that hooks (like "on %done") of imported types are not executed.

Here a simple test :

test.pac2

module Test;

import BinPAC;
import p;

export type T = unit(){

  d : ABC::A;

  on %done{
    print self.d;
  }
};

p.pac2

module ABC;

import BinPAC;

type A = unit(){
  var c : uint<8> = 3;
  var a : bytes;
  b : uint<8> {self.c = 4;}

  on %done{
    self.a = b"TEST";
    print "AAA";
  }
};

echo "A" | pac-driver test.pac2
<c=3, b=65>

We notice that "c" is not modified and "a" is not defined.

Starting from tests/binpac/enum/compare-enum.pac2, in the following code, the comparision of an integer (not converted!) to an enum label seems to return equal in any case.

# @TEST-EXEC: printf '\101\102' | pac-driver-test %INPUT >output
# @TEST-EXEC: btest-diff output
#
# @TEST-KNOWN-FAILURE: there's a problem with comparing enum labels.

module Mini;

type ObjectType = enum {
  DataType = 0x41,
  EndOfData = 0x43
};

if ( ObjectType::DataType == ObjectType::DataType ) {
  print "equal";
}

if ( ObjectType::DataType != ObjectType::EndOfData ) {
  print "not equal";
}

export type test = unit {
#  a : uint8 &convert=ObjectType($$);
#  b : uint8 &convert=ObjectType($$);
  a : uint8;
  b : uint8;

  on %done {
    if ( self.a == ObjectType::DataType )
      print "equal";
    if ( self.b != ObjectType::DataType )
      print "not equal";
  }
};

This prints

equal
not equal
equal

and the final "not equal" is missing.

As far as I understand it, either should the comparision work the right way or the compiler say that it is not supported.

I forgot to add that the comparision works correctly if I cast the enum label to an int before, like

if ( self.a == cast<uint8>(ObjectType::DataType) )

By the way, the unmodified test case seems to run correctly, despite it's marked as known failure?

Hello,

I'm trying to compile Hilti on Debian 9 (fresh install). I have all the required dependencies, compiled and installed from their source. I've ran into several problems that I managed to solve, but now I'm stuck.

At 91% completion, I have this error:

[ 91%] Generating autogen/spicy.hlt
error, cannot find input file /home/gmonarque/documents/hilti/libspicy/spicy.spicy

Here is the full log: https://pastebin.com/zqSkS21v

The file spicy.spicy is not missing from the directory. Can you please help? I don't know where to look at to solve this. Also, if you have any advice on how to properly compile this on Debian, I'm all ears :) Thanks!

Hello !

I am currently testing CPU performance of parsers and i think found an improvement for &unti spicy attribute.

Here a representative parser of my test :

module Foo;

export type Test = unit {
  data : bytes &until("\x00");
};

I notice the processing time is quite sensitive (nearly x2) to the payload size because of &until("\x00") when "\x00" delimiter is not present.

I see that the &until attribute is translated into unpack (data, cur) Hilti::Packed::BytesDelim "\x00" which is transformed into a llvm loop composed of llvm basic block.
https://github.com/rsmmr/hilti/blob/master/hilti/codegen/unpacker.cc#L134-L210

At each loop step tree functions are called:

• hlt_iterator_bytes_eq
• hlt_bytes_match_at
• hlt_iterator_bytes_incr

The pseudo code is :

while 1 {
  if ( hlt_iterator_bytes_eq(cur, END_BYTE))
     //  => raise Hilti::WouldBloc
     break;
  else{
    if hlt_bytes_match_at(cur, delim)
      // => Find !
      break;
    else
      hlt_iterator_bytes_incr(cur) # cur = cur+1
  }
}

Even through these functions are "inlines", the loop assembly is quite heavy to just search a string pattern in a bytes.

I saw that __hlt_bytes_find_byte_from [EDIT : __hlt_bytes_find_bytes] exists to search a string pattern, it is written in C and use memchr of libc to search a char. memch seems to be optimized for performance (better than a simple loop on each char)
https://github.com/rsmmr/hilti/blob/master/libhilti/bytes.c#L1019-L1029
http://code.metager.de/source/xref/gnu/glibc/string/memchr.c

Before trying to use __hlt_bytes_find_bytes for &until i would like to know if there is maybe a reason that it is not used in the unpacker.cc ?

# btest -d spicy/unit/wrong-id.spicy
[ 66%] spicy.unit.wrong-id-3 ... failed
  % 'btest-diff output' failed unexpectedly (exit code 1)
  % cat .diag
  == File ===============================
  <no location>: internal error, $$ not bound in CodeBuilder::visit(expression::ParserState* p)
  Aborted
  == Diff ===============================
  --- /tmp/test-diff.3994.output.baseline.tmp   2017-01-13 10:54:52.981797595 +0000
  +++ /tmp/test-diff.3994.output.tmp    2017-01-13 10:54:53.031760487 +0000
  @@ -1 +1,2 @@
  <no location>: internal error, $$ not bound in CodeBuilder::visit(expression::ParserState* p)
  +Aborted
  =======================================

  % cat .stderr

1 of 3 tests failed

Tested in the rsmmr/hilti Docker image:

# spicy-config --version
Spicy 0.5-4

I'm building hilti with bro following the instructions on the web site:
http://www.icir.org/hilti/install/index.html

I'm building on Ubuntu 14.04 with clang 3.9.

I've successfully built bro. I've run the bro tests and all passed.

During the hilti build I get the following error:

In file included from /home/user1/hilti/bro/src/Converter.cc:3:
/home/user1/hilti/bro/src/Plugin.h:50:71: error: non-virtual member function marked 'override' hides virtual member function
int HookLoadFile(const std::string& file, const std::string& ext) override;
^
/home/user1/bro/src/plugin/Plugin.h:644:14: note: hidden overloaded virtual function 'plugin::Plugin::HookLoadFile' declared here:
different number of parameters (3 vs 2)
virtual int HookLoadFile(const LoadType type, const std::string& file, const std::string& resolved);

If I try to compile the following minimal grammar with
hilti-build -o a.out tools/pac-driver/pac-driver.cc libbinpac/parsers/minimal.pac2, and get an error message from hiltic, see below. If I do not use convert but the line without it (see code), the compilation terminates successfully. Additionally, when running the grammar with pac-driver's JIT, there is no problem with convert.

Minimal grammar:

module Profinet;

import BinPAC;

type EtherType = enum {
  IPV4 = 0x800,
  ARP = 0x806,
  Wake_on_LAN = 0x842
};

export type EthernetPacket = unit {
  destination: bytes &length=6;
  source: bytes &length=6;
  ethertype: uint<16> &convert=EtherType($$);
#  ethertype: uint<16>;
  payload: bytes &eod;

  on %done {
    print self.ethertype;
  }
};

Error message:

>>> Undef
profinet.hb3547.tmp.hlt:714-714: error, unknown ID Undef [pass::hilti::IdResolver]
profinet.hb3547.tmp.hlt: Aborting due to verification error.
error running hiltic, aborting

Part of profinet.hb3547.tmp.hlt:

  @if_true_23
    struct.set __self "ethertype" Undef
    jump @if_cont_23

I tried the included grammars pcap.pac2 and tftp.pac2, they have the same problem with hilti-build, but not with pac-driver.