gdslang / gdsl-toolkit

Generic Decoder Specification Language Toolkit

GDSL Toolkit

The GDSL toolkit is a software framework for specifying machine language decoders. It is meant to allow a natural specification that resembles the way the vendor manuals describe the instruction set. The toolkit comes with a set of decoders, semantics translators to the analysis-friendly intermediate reprensentation RReil, IR optimizations, and bindings to other programming languages such as C++ and Java.

Dependencies

This section describes how the GDSL toolkit is built and which dependencies are required. The GDSL compiler and core libraries require the following software to be on the current search PATH:

• MLton
• CMake
• A C11 and a C++11 compiler

Furthermore, the following software packages and features are required by some of the optional demo applications:

• Libelf (http://www.mr511.de/software/) [>= version 0.152]

Building the Toolkit

The toolkit can be built using CMake:

cd gdsl-toolkit/
mkdir build
cd build/
cmake ..
make

This command will build the toolkit for all available architectures. To build a specific architecture, specify the respective target:

...
make gdsl_x86_rreil

Here, we build the x86 frontend. Similarly, targets for specific libraries or tools can be specified.

Demos and Libraries

The GDSL toolkit contains a number of demo applications and libraries. The tools and libraries are presented in wiki (http://code.google.com/p/gdsl-toolkit/).

Roadmap

• Architectures
  • Full support for ARMv7 and ARMv8
• Testing
  • Support testing of all our architectures (reusing common parts of the tester)
  • Testing of the decoder by reacting to / expecting illegal instruction exceptions

gdsl-toolkit's Issues

Compiler bug, multiple default cases.

What steps will reproduce the problem?
1. Compile the code at head: 72bd62ac73a981b9313e317d853e7f57b9256c38


gdsl-arm7.c: In function '_slash_':
gdsl-arm7.c:849:9: error: multiple default labels in one switch
         default: {
         ^
gdsl-arm7.c:820:9: error: this is the first default label
         default: {
         ^

Original issue reported on code.google.com by [email protected] on 8 Aug 2014 at 9:54

• Merged into: #10

Including both the front-end specific and the multiplexing header

Including both the front-end-specific and the multiplexing header currently requires the programmer to obey a specific inclusion order - the front-end specific header version is required to be included before the generic header.

Header files should not put dependencies on one another. Every inclusion order should be valid.

Error Building tools

I'm trying to build the tools under /tools. (semantics-opt, etc). I've built and installed gdsl successfully so far. If I cd into tools and run make, I get the following:

gcc -c -g3 -std=gnu11 -pedantic -Wall -Wfatal-errors -DRELAXEDFATAL -DRELAXEDFATAL -I../include semantics-cli.c -o semantics-cli.o
semantics-cli.c:10:18: fatal error: gdsl.h: No such file or directory
 #include <gdsl.h>
                  ^
compilation terminated.
make: *** [semantics-cli.o] Error 1

Checking in the ../include directory, gdsl.h is a broken symlink to ../gdsl_manual.h. Is there something I'm missing about how to build the tools?

Thanks

32-bit Mode

Is 32-bit mode still supported for x86? I'm trying to use the cppgdsl interface and I'd like to dynamically invoke the x86-rreil frontend in either 32-bit of 64-bit mode but I don't see a way to configure it. Thanks!

Question: Size of the right hand side of an assignment

When I translate a X86-32 instruction IMUL EAX, DWORD PTR SS:[EBP+8] (code: "\x0f\xaf\x45\x08") into RREIL, I got:

IP =:32 (IP + 4)
T0 =:64 sx[32]A
T2 =:64 zx[32](BP + 8)
T1 =:32 *[64](SS_Base + T2)
T1 =:64 sx[32]T1
T3 =:64 T0 * T1
T4 =:33 sx[1]T3.63
FLAGS.11 =:1 T3.31 !=:33 T4
FLAGS =:1 FLAGS.11
FLAGS.7 =:1 arbitrary
FLAGS.6 =:1 arbitrary
FLAGS.4 =:1 arbitrary
FLAGS.2 =:1 arbitrary
LEU =:1 FLAGS | FLAGS.6
LTS =:1 FLAGS.7 !=:1 FLAGS.11
LES =:1 LTS | FLAGS.6
A =:32 T3

At the last assignment statement, the left A is 32 bit, while the right T3 is 64 bit.
Is this correct behavior?

implemented RReil grammar is cannot express sizes for comparisons

The current RReil grammar does not match the one published in the APLAS paper. 
We use a very unintuitive short-cut that avoids storing the size of a 
comparison in the ```sexpr```.

The statement

{{{
FLAGS :=1 x <=u8 y
}}}

that should assign the 1-bit result of the 8-bit unsigned comparison in FLAGS 
uses the size field of the assignment to store the size of the comparison (i.e. 
internally the assignment above is stored as

{{{
FLAGS :=8 x <=u y
}}}

and the comparison has no size field). This is not only counter-intuitive but 
it prevents us from writing

{{{
cbranch x <=u8 y ? tgt1 : tgt2
}}}

since ```cbranch``` does not have a size field where we could store 8. Thus, we 
should have a size field in the comparison rule of ```sexpr`` and use the 
intuitive semantics in that we store 1 for the size of the assignment for 
comparisons.

Original issue reported on code.google.com by [email protected] on 14 Nov 2014 at 6:53

Parameter-less functions

GDSL type declarations like

type asm_signedness_callbacks = {
  asm_signed: (()) -> signedness_obj,
  asm_unsigned: (()) -> signedness_obj
}

together with calls such as

cbs.signedness.asm_signed void

result in C structures that contain functions that expect zero arguments:

typedef struct {
  obj_t (*asm_signed)();
  obj_t (*asm_unsigned)();
} unboxed_asm_signedness_callbacks_t;

Any GDSL function should, however, expect at least the state_t object to be 
passed to it.

Original issue reported on code.google.com by [email protected] on 25 Jul 2014 at 9:37

Readme and License are outdated

The Readme and License files are outdated.

Original issue reported on code.google.com by [email protected] on 24 Oct 2013 at 1:12

Missing newline after "block"

[specifications/rreil/rreil-translator.ml:138.1-144.1] Error: duplicate 
variable declaration decode-translate-super-blockunhandled exception: 
CompilationError

Original issue reported on code.google.com by [email protected] on 15 Sep 2014 at 2:36

Heap overflow in readhex_hex_read function (libs/readhex/src/readhex.c)

>What steps will reproduce the problem?
Write more than 32 characters at once whenever readhex_hex_read is called (as 
for example in the tool semantics-cli)
echo 
"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAA" | ./semantics-cli

>What is the expected output? What do you see instead?
The expected output is that the tool will decode the hex values into the GDSL 
instructions.

>What version of the product are you using? On what operating system?
I'm using the current version in the repository on 64-bit Ubuntu Linux 13.04.

>Please provide any additional information below.
This issue is caused because the if statement on line 40 of 
libs/readhex/src/readhex.c checks the variable size_str (which is always the 
value 2) instead of the variable length (which is used to access the buffer) to 
determine whether it should realloc the buffer.  Changing this will fix the 
issue.

Original issue reported on code.google.com by [email protected] on 11 Nov 2013 at 6:01

C Structure Member Ordering

The members of C structures generated by the GDSL compiler are not ordered. 
Ordering is, however, necessary for gdsl-multiplex library to work correctly.

Original issue reported on code.google.com by [email protected] on 25 Jul 2014 at 1:58

Compilation error on 32bit Linux using Gcc-4.8.2

The toolkit does not compile on a 32bit Linux using Gcc-4.8.2. The following 
compiler error is produced:

./libs/rreil-sim/src/simulator/ops.c:133:1: error: unknown type name
'__uint128_t'

Original issue reported on code.google.com by [email protected] on 24 Oct 2013 at 1:16

Incorrectly Prints X86 Instructions with a negative Immediate value

>What steps will reproduce the problem?
The problem can be illustrated by running the semantics-cli tool.  The last 
character of the immediate operand is missing.
$ echo "48 8d 45 ff" | ./semantics-cli
Configuration:
        mode64: true
        default_opnd_sz = 32: true
[48 8d 45 ff] LEA RAX, PTR(0) [RBP+(-)@3] 
---------------------------
FLAGS.9 =:1 1
T0 =:64 zx[64->64](BP + (-))
A =:64 T0

>What is the expected output? What do you see instead?
Instead it should print the entire immediate, like so:
Configuration:
        mode64: true
        default_opnd_sz = 32: true
[48 8d 45 ff] LEA RAX, PTR(0) [RBP+(-1)@3] 
---------------------------
FLAGS.9 =:1 1
T0 =:64 zx[64->64](BP + (-1))
A =:64 T0

>What version of the product are you using? On what operating system?
I'm using the current version in the repository on Ubuntu Linux 13.04.

>Please provide any additional information below.
The offending code is in the int_to_string function in gdsl-x86-rreil.c on 
lines 480-481.  Instead of the lines:
str[negate + size - 1] = ')';
str[negate + size] = 0;
The lines should be:
str[negate + size] = ')';
str[negate + size + 1] = 0;

Original issue reported on code.google.com by [email protected] on 12 Nov 2013 at 3:20

decode() returns struct2_t

decode() returns a struct2_t structure; it needs to return an insndata_t 
instance.

Original issue reported on code.google.com by [email protected] on 1 Aug 2014 at 1:06

X86 "MOV r/m64, imm32" not sign extended as expected

>What steps will reproduce the problem?
$ echo '49 c7 c2 ff ff ff ff' | /semantics-cli
Configuration:
        mode64: true
        default_opnd_sz = 32: true
[49 c7 c2 ff ff ff ff] MOV R10, 4294967295@3 
---------------------------
FLAGS.9 =:1 1
R10 =:64 4294967295

>What is the expected output? What do you see instead?
$ echo '49 c7 c2 ff ff ff ff' | /semantics-cli
Configuration:
        mode64: true
        default_opnd_sz = 32: true
[49 c7 c2 ff ff ff ff] MOV R10, 4294967295@3 
---------------------------
FLAGS.9 =:1 1
R10 =:64 sx[32->64]4294967295
or
R10 =:64 18446744073709551615 (i.e. the value of 0xffffffffffffffff)

>What version of the product are you using? On what operating system?
I'm using the latest version in the repository on 64-bit Ubuntu 13.04.

>Please provide any additional information below.
According to the Intel Software Architectures Software Development Manual 
Volume 2, REX.W + C7 /0 io is supposed to be decoded to "MOV r/m64, imm32" with 
the imm32 sign extended to 64-bits.  However, it isn't sign extended in the 
semantics-cli tool.

Original issue reported on code.google.com by [email protected] on 17 Nov 2013 at 10:39

Error message when "no decoder" should be clearer.

What steps will reproduce the problem?
1. Write a specification with missing "pretty" exported functions
2. run demo application

What is the expected output? What do you see instead?
See: "GDSL modules contain no decoder function"
Would like to see: "GDSL module is missing decoder function pretty : (insndata) 
-> rope"

What version of the product are you using? On what operating system?
git master
linux

Please provide any additional information below.
Not a critical issue as most people won't be writing new decoders, but it would 
have saved me a lot of time searching for the answer.

Original issue reported on code.google.com by [email protected] on 31 Jul 2014 at 3:46

The expression 'not (8 > 0 or 42 > 0)' is compiled incorrectly by the GDSL compiler

The expression 'not (8 > 0 or 42 > 0)' is compiled incorrectly by the GDSL compiler. Consider the following GDSL program:

export test_right : (int) -> |1|
export test_wrong : (int) -> |1|

val test_right x = not (8 > 0)
val test_wrong x = not (8 > 0 or 42 > 0)

The code is compiled into the C code shown below:

...
/* test_right */
static int_t test_right(state_t s,int_t x) {
  return vec_not(s,gen_vec(1, 0<8)).data;
}
/* test_wrong */
static int_t test_wrong(state_t s,int_t x) {
  return vec_not(s,gen_vec(0, (0<8) | (0<42))).data;
}
...

Here, the size of the vector is 1 in case of the test_right function, but 0 in case of test_wrong function. The size of the vector should be 1 in both cases.

Library cppgdsl lin_scale has no to_string implementation

The include file for lin_scale in the cppgdsl library references a to_string method. However the lin_scale cpp file (libs/cppgdsl/src/rreil/linear/lin_scale.cpp) does not have an implementation for this method. However, the class that lin_scale inherits from (linear) implements this method.

The problem comes from the compiler trying to fetch the lin_scale's implementation, which does not exist. Thus the following error is generated:

classifier.o: In function `the_visitor::visit(gdsl::rreil::lin_scale*)':
./example.cpp:68: undefined reference to `gdsl::rreil::lin_scale::to_string()'

I was able to work around the issue by removing the following line from lin_scale.h:

36   std::string to_string();

Errors with Python and Java Bindings

I want to use GDSL to translate x86 machine code to RREIL then do my own separate analysis on the RREIL, what would be the path of lease resistance to getting access to the RREIL? I see Python and Java bindings. The python bindings give a build error:

python setup.py build_ext --inplace
running build_ext
building '_pygdsl' extension
swigging pygdsl.i to pygdsl_wrap.c
swig -python -o pygdsl_wrap.c pygdsl.i
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DHAVE_CONFIG_H=1 -DGDSL_X86=1 -I../ -I./ -I/usr/include/python2.7 -c pygdsl_wrap.c -o build/temp.linux-x86_64-2.7/pygdsl_wrap.o -std=c11
pygdsl_wrap.c: In function ‘init_pygdsl’:
pygdsl_wrap.c:4142:87: error: ‘CON_SEM_PRESERVATION_EVERYWHERE’ undeclared (first use in this function)
   SWIG_Python_SetConstant(d, "GDSL_SEM_PRESERVATION_EVERYWHERE",SWIG_From_long((long)(CON_SEM_PRESERVATION_EVERYWHERE)));
                                                                                       ^
pygdsl_wrap.c:4142:87: note: each undeclared identifier is reported only once for each function it appears in
pygdsl_wrap.c:4143:82: error: ‘CON_SEM_PRESERVATION_BLOCK’ undeclared (first use in this function)
   SWIG_Python_SetConstant(d, "GDSL_SEM_PRESERVATION_BLOCK",SWIG_From_long((long)(CON_SEM_PRESERVATION_BLOCK)));
                                                                                  ^
pygdsl_wrap.c:4144:84: error: ‘CON_SEM_PRESERVATION_CONTEXT’ undeclared (first use in this function)
   SWIG_Python_SetConstant(d, "GDSL_SEM_PRESERVATION_CONTEXT",SWIG_From_long((long)(CON_SEM_PRESERVATION_CONTEXT)));
                                                                                    ^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
make: *** [_pygdsl.so] Error 1

The java bindings have built and appear to work, but the example program fails because Gdsl.getFrontends() returns an empty array. I think it's looking in a directory but I haven't found any documentation on where and what I should be configuring so it can find a frontend.

Thanks

Compilation error with spaces in bit strings.

What steps will reproduce the problem?
1. Spaces in bit strings cause compilation errors.
2. val / ['/cond 0001 0010 '] errors
   val / ['/cond 00010010 '] doesn't

What is the expected output? What do you see instead?
Code should compile, or throw parse error.

instead 

gdsl-arm7.c: In function '_slash_':
gdsl-arm7.c:553:9: error: multiple default labels in one switch
         default: {
         ^
gdsl-arm7.c:508:9: error: this is the first default label
         default: {


What version of the product are you using? On what operating system?
git master
Linux

Original issue reported on code.google.com by [email protected] on 30 Jul 2014 at 11:38

What is gdsl-toolkit?

From the beginning of README:

GDSL Toolkit
This section describes how the GDSL toolkit is built and which dependencies are required.

Thanks you very much for describing that. But can you start your README with the information about what "GDSL Toolkit" is in the first place?

Support for AVX-512 instructions is missing

The support for newly added AVX-512 instructions is missing in the x86 decoder specification of GSL. See https://software.intel.com/en-us/blogs/2013/avx-512-instructions.

clock_gettime found in librt which is not linked

What steps will reproduce the problem?
1. Try to build x86 frontend on Gentoo AMD64

What is the expected output? What do you see instead?

<snip>gdsl/gdsl-toolkit/./tools/x86-test-stats-runner/src/main.c:282: undefined 
reference to `clock_gettime'
collect2: error: ld returned 1 exit status

What version of the product are you using? On what operating system?

Gentoo AMD64 latest

Please provide any additional information below.

Add the following to configure.ac to fix the issue:

AC_SEARCH_LIBS([clock_gettime], [rt posix4])

Original issue reported on code.google.com by [email protected] on 21 Jan 2014 at 4:50

Missing dependency in installation instructions: ml-ulex

The installation instructions do not ml-ulex as a dependency. The program is 
provided by ml-lpt package on Debian Linux.

Original issue reported on code.google.com by [email protected] on 24 Oct 2013 at 1:19