google / vxsig

Automatically generate AV byte signatures from sets of similar binaries.

VxSig

Copyright 2011-2024 Google LLC

Disclaimer: This is not an official Google product (experimental or otherwise), it is just code that happens to be owned by Google.

Introduction

VxSig is a tool and library to automatically generate AV byte signatures from sets of similar binaries. It processes files generated by BinExport and BinDiff.

Signatures can be generated for Yara (the default) and ClamAV.

Status

VxSig is a mature tool that has been used at Google to create signature and scan for many kinds of malware and targetted threats.

Quick Start

VxSig uses Bazel to build and manage its dependencies. The preferred way to use a current version is via Bazelisk, so install that first. For example, on Debian-based Linux distributions do:

(cd /tmp && \
  wget -qO- \
  https://github.com/bazelbuild/bazelisk/releases/download/v1.19.0/bazelisk-linux-$(dpkg --print-architecture) \
  > bazelisk && \
  echo 'd28b588ac0916abd6bf02defb5433f6eddf7cba35ffa808eabb65a44aab226f7  bazelisk' | \
  sha256sum -c && \
  chmod +x bazelisk && \
  sudo mv bazelisk /usr/local/bin/ \
)

Refer to the Bazel Getting started guide for how to get started on other platforms.

Clone and run the build:

git clone https://github.com/google/vxsig && cd vxsig
bazelisk build -c opt //vxsig:vxsig

To build an example Yara signature:

bazel-bin/vxsig/vxsig --detection_name=VxSigTestSig --trim_length=400 \
  vxsig/testdata/592fvs2065.BinDiff

The output should look like this (truncated):

----8<--------8<---- Signature ----8<--------8<----                    
rule VxSigTestSig {
  meta:
    vxsig_build = "redacted"
  strings:
    $ = {
         00008bd85985db5975
         // 00401049: mov ebx, eax
         // 0040104b: pop ecx
         // 0040104c: test ebx, ebx
         // 0040104e: pop ecx
         // 0040104f: jnz 0x4010b7
      [-]110000435653e8
         // 004010c0: inc ebx
         // 004010c1: push esi
         // 004010c2: push ebx
         // 004010c3: call 0x40226c
      [-]1100006a10be
         // 004010fe: push b1 0x10
         // 00401100: mov esi, 0x4042a8
      [-]6a0056e8
         // 00401105: push b1 0x0
         // 00401107: push esi
         // 0040110b: call 0x402266
...

Further reading / Similar tools

• The original thesis that provided the basis for this tool (German language only): Automatisierte Signaturgenerierung für Malware-Stämme
• zynamics VxClass, a discontinued malware analysis pipeline using a previous version of VxSig.
• Cisco's Talos Group's BASS Automated Signature Synthesizer, an open-source reimplementation of the thesis
• functionsimsearch, a tool that can be used to create a corpus of files for computing function occurrence counts.

Getting Involved

If you want to contribute, please read CONTRIBUTING.md and send pull requests. You can also report bugs or file feature requests.

vxsig's Issues

How to generate signatures for a set of binaries (more than two binaries)

Hi all,

Thank you for your wonderful work. I am trying to use VxSig to generate AV signatures for a few malware families, where each family consists of multiple binaries (>2).

If I understand correctly, VxSig takes a pair of binaries as input, locating the common parts, and finally generate YARA rules to represent them.

Now how to generate signatures for a pair of binaries is clear. However, I am still confused about how to use VxSig to generate YARA rules for a set of binaries (more than two binaries). Actually I have some initial ideas, but I am not sure.

Could you give me some suggestions, or, what is the best practice of dealing with more than two binaries?

Much thanks in advance!

VxSig doesn't seem to be producing correct Yara signatures when using static binaries

The Yara signatures look off, and don't correctly match the expected binaries when using static binaries, produced using Ida Free, BinExport, and processed via VxSig.

Processing the following files outputs the warnings:
string "$" may slow down scanning
warning: rule "VxSig_Signature": too many matches for $, results for this rule may be incorrect
warning: rule "VxSig_Signature": too many matches for $, results for this rule may be incorrect
warning: rule "VxSig_Signature": too many matches for $, results for this rule may be incorrect
warning: rule "VxSig_Signature": too many matches for $, results for this rule may be incorrect
warning: rule "VxSig_Signature": too many matches for $, results for this rule may be incorrect

And only one of the two files used to produce the diff is matched by the signatures.

Files attached:
vxsig_attempt.zip

Build is broken on bazel version >= 7, 6.5.0 is the last version that works. There's also a dependency on compiler version.

To successfully build without protobuf redefinition errors, prefix the build statement with something similar to:

USE_BAZEL_VERSION=6.5.0 CC=gcc-9

BinExport files misformed

When using last commit (470f04635c4bb8007035bf1fa4f529b80f6e6d3b), some BinExport files in are invalid in the testdata directory.

List of invalid files (in vxsig/testdata):

• 328b26dc3f0d8543e151495f4d6f3960323e3f51223522c2e4cd1e2fe9f9ed8f.BinExport
• 8433c9a6345d210d2196096461804d7137bbf2a6b71b20cc21f4ecf7d15ef6c2.BinExport
• 61971471cedcb4daed8d07ad79297568ffdaa17eb4ff301dc953cfafa91a4507.BinExport
• sshd.korg.BinExport

The tool used with any of the associated BinDiff file will fail with this error message.

± % bazel-bin/vxsig/vxsig --detection_name=VxSigTestSig --trim_length=400 vxsig/testdata/sshd.korg_vs_sshd.trojan1.BinDiff
Parsing diff results
Loading function metadata and instruction data
[siggen_main.cc : 102] RAW: Check status.ok() failed: Failed to generate signature: failed parsing vxsig/testdata/sshd.korg.BinExport
[1]    13700 abort      bazel-bin/vxsig/vxsig --detection_name=VxSigTestSig --trim_length=400 

Remark : The others files works and a signature is generated.

The list of invalid files was found using this :

find vxsig/testdata -name '*.BinExport' -exec sh -c "cat {} | protoc --decode_raw 1>/dev/null || echo {}"

Compilation Failure: No filesystem.cc file

Dear folks,

May I have your advice over this error? Thank you so much. Stay safe from virus.

Regards,
Anthony

hacker@ubuntu:$ git clone https://github.com/google/vxsig && cd vxsig
Cloning into 'vxsig'...
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 304 (delta 7), reused 10 (delta 2), pack-reused 271
Receiving objects: 100% (304/304), 18.97 MiB | 5.48 MiB/s, done.
Resolving deltas: 100% (176/176), done.
hacker@ubuntu:/vxsig$ bazel build -c opt //vxsig:vxsig
Starting local Bazel server and connecting to it...
INFO: Analyzed target //vxsig:vxsig (38 packages loaded, 944 targets configured).
INFO: Found 1 target...
INFO: Deleting stale sandbox base /home/hacker/.cache/bazel/bazel_hacker/2d6676569272affd55cecfcb749b2caa/sandbox
INFO: From Compiling vxsig/siggen_main.cc:
In file included from ./vxsig/match_chain_table.h:53:0,
from ./vxsig/generic_signature.h:22,
from ./vxsig/siggen.h:37,
from vxsig/siggen_main.cc:32:
bazel-out/k8-opt/bin/vxsig/vxsig.pb.h: In function 'int main(int, char**)':
bazel-out/k8-opt/bin/vxsig/vxsig.pb.h:2640:19: warning: 'trim_algorithm' may be used uninitialized in this function [-Wmaybe-uninitialized]
trim_algorithm = value;

vxsig/siggen_main.cc:58:47: note: 'trim_algorithm' was declared here
SignatureDefinition::SignatureTrimAlgorithm trim_algorithm;
                                            ^~~~~~~~~~~~~~
ERROR: /home/hacker/.cache/bazel/_bazel_hacker/2d6676569272affd55cecfcb749b2caa/external/com_google_binexport/BUILD.bazel:100:11: C++ compilation of rule '@com_google_binexport//:filesystem' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 52 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
external/com_google_binexport/util/filesystem.cc:39:10: fatal error: filesystem: No such file or directory
#include <filesystem>
       ^~~~~~~~~~~~
compilation terminated.
Target //vxsig:vxsig failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 41.416s, Critical Path: 7.95s
INFO: 17 processes: 17 linux-sandbox.
FAILED: Build did NOT complete successfully

Error generating signatures

Some of the bindiff files from testdata are creating error while generating signatures with vxsig. For example, this one https://github.com/google/vxsig/blob/master/vxsig/testdata/61971471cedcb4daed8d07ad79297568ffdaa17eb4ff301dc953cfafa91a4507_vs_8433c9a6345d210d2196096461804d7137bbf2a6b71b20cc21f4ecf7d15ef6c2.BinDiff
Some of them are good to go.

Here's the error shown below

• Its also creating this error for bindiff files, generated using Bindiff 5.

build error on windows: xargs: command not found

bazel build -c opt //vxsig:vxsig
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Repository com_google_binexport instantiated at:
  D:/test/vxsig/WORKSPACE:50:13: in <toplevel>
Repository rule http_archive defined at:
  C:/users/user123456/_bazel_admin/4uuzjk7y/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'com_google_binexport':
   Traceback (most recent call last):
        File "C:/users/user123456/_bazel_admin/4uuzjk7y/external/bazel_tools/tools/build_defs/repo/http.bzl", line 143, column 10, in _http_archive_impl
                patch(ctx, auth = auth)
        File "C:/users/user123456/_bazel_admin/4uuzjk7y/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 193, column 21, in patch
                fail("Error applying patch command %s:\n%s%s" %
Error in fail: Error applying patch command find . -path ./third_party -prune -o \( -name '*.cc' -o -name '*.h' \) -print0 |xargs -0 -P8 -n1 sed -i.bak 's,^\(#include "\)third_party/\(absl\),\1\2,g':
FIND: ������ʽ����ȷ
/usr/bin/bash: line 1: xargs: command not found
ERROR: D:/test/vxsig/WORKSPACE:50:13: fetching http_archive rule //external:com_google_binexport: Traceback (most recent call last):
        File "C:/users/user123456/_bazel_admin/4uuzjk7y/external/bazel_tools/tools/build_defs/repo/http.bzl", line 143, column 10, in _http_archive_impl
                patch(ctx, auth = auth)
        File "C:/users/user123456/_bazel_admin/4uuzjk7y/external/bazel_tools/tools/build_defs/repo/utils.bzl", line 193, column 21, in patch
                fail("Error applying patch command %s:\n%s%s" %
Error in fail: Error applying patch command find . -path ./third_party -prune -o \( -name '*.cc' -o -name '*.h' \) -print0 |xargs -0 -P8 -n1 sed -i.bak 's,^\(#include "\)third_party/\(absl\),\1\2,g':
FIND: ������ʽ����ȷ
/usr/bin/bash: line 1: xargs: command not found
ERROR: D:/test/vxsig/vxsig/BUILD.bazel:438:10: //vxsig:vxsig depends on @com_google_binexport//:filesystem in repository @com_google_binexport which failed to fetch. no such package '@com_google_binexport//': Error applying patch command find . -path ./third_party -prune -o \( -name '*.cc' -o -name '*.h' \) -print0 |xargs -0 -P8 -n1 sed -i.bak 's,^\(#include "\)third_party/\(absl\),\1\2,g':
FIND: ������ʽ����ȷ
/usr/bin/bash: line 1: xargs: command not found
ERROR: Analysis of target '//vxsig:vxsig' failed; build aborted: Analysis failed
INFO: Elapsed time: 35.611s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (37 packages loaded, 124 targets configured)

Fail to build vxsig, absl header not found

Here's the error shown below.
Please help to fix the problem.
Thanks,
Glen

$ bazel version
Build label: 1.0.0

$ git clone https://github.com/google/vxsig
$ cd vxsig

$ git log
commit 27f5ef3 (HEAD -> master, origin/master, origin/HEAD)

$ bazel build -c opt //vxsig:vxsig --incompatible_disable_deprecated_attr_params=false

In file included from external/com_google_binexport/util/canonical_errors.cc:15:
bazel-out/darwin-opt/bin/external/com_google_binexport/_virtual_includes/status/third_party/zynamics/binexport/util/canonical_errors.h:18:10: fatal error: 'third_party/absl/base/attributes.h' file not found
#include "third_party/absl/base/attributes.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target //vxsig:vxsig failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 158.249s, Critical Path: 18.96s
INFO: 232 processes: 232 darwin-sandbox.
FAILED: Build did NOT complete successfully

Add command-line options to control signature output

VxSig should be able to output:

• just the signature to stdout (no messages or messages to stderr)
• write signatures to file in the requested format

Cutting and pasting from stdout is a bit tedious.

Replace usages of `std::set<T>` with `absl::flat_hash_set<T>` / `absl::bree_set<T>`

This should speed up the common subsequence operations a bit