aquasecurity / tracee

Linux Runtime Security and Forensics using eBPF

Before moving on, please consider giving us a GitHub star ⭐️. Thank you!

About Tracee

Tracee is a runtime security and observability tool that helps you understand how your system and applications behave.
It is using eBPF technology to tap into your system and expose that information as events that you can consume.
Events range from factual system activity events to sophisticated security events that detect suspicious behavioral patterns.

To learn more about Tracee, check out the documentation.

Quickstart

To quickly try Tracee use one of the following snippets. For a more complete installation guide, check out the Installation section.
Tracee should run on most common Linux distributions and kernels. For compatibility information see the Prerequisites page. Mac users, please read this FAQ.

Using Docker

docker run --name tracee -it --rm \
  --pid=host --cgroupns=host --privileged \
  -v /etc/os-release:/etc/os-release-host:ro \
  -v /var/run:/var/run:ro \
  aquasec/tracee:latest

For a complete walkthrough please see the Docker getting started guide.

On Kubernetes

helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
helm install tracee aqua/tracee --namespace tracee --create-namespace

kubectl logs --follow --namespace tracee daemonset/tracee

For a complete walkthrough please see the Kubernetes getting started guide.

Contributing

Join the community, and talk to us about any matter in the GitHub Discussions or Slack.
If you run into any trouble using Tracee or you would like to give use user feedback, please create an issue.

Find more information on contribution documentation.

More about Aqua Security

Tracee is an Aqua Security open source project.
Learn about our open source work and portfolio here.

tracee's Issues

chmod trace bug

command:

chmod +x test

output:

17431.801997   ubuntu           4026531840   4026531836   1000   fchmodat         chmod            19716  19716  10285  0           -100 test invalid file type|S_IRWXU|S_IRWXG|S_IROTH|S_IXOTH -1 

notice the "invalid file type" in mode

embed the ebpf c program in the go binary

this will make distribution more reliable

Rename start.py to tracee.py

So that it's more obvious that Tracee is what we are running, if we show it in a demo

CWD is sometimes changed to kernel headers directory

(reported by Idan)
In a go program that loads tracee as a library and writes to a file simultaneously, meaning there are 2 goroutins, one that runs tracee, another that writes files to a relative path. Some files are being written to the kernel headers directory, instead of the directory where this program runs. The hunch is it's related to ebpf compilation, probably something there is cding around, and the process is affected.

Missing clone flags

Clone syscall flags are currently missing from the output.
Need to add them

Add automated code coverage

We should use codecov on Tracee like most of our other projects

Don't copy executable multiple times

When capture executables is enabled, every file is copied on each exec.
This wastes memory, as the same file can be copied multiple times.
If the binary was already copied (check using access + stat ctime and cache results), don't copy it

Error running as container on Ubuntu 20.04

root@vagrant:~# uname -a
Linux vagrant 5.4.0-31-generic #35-Ubuntu SMP Thu May 7 20:20:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@vagrant:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal
root@vagrant:~# docker run --name tracee --rm --privileged -v /lib/modules/:/lib/modules/:ro -v /usr/src:/usr/src:ro aquasec/tracee:latest
sh: 1: modprobe: not found
chdir(/lib/modules/5.4.0-31-generic/build): No such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5333c6]

goroutine 1 [running]:
github.com/iovisor/gobpf/bcc.(*Module).Load(0x0, 0xc00008ab30, 0xe, 0x2, 0x0, 0x0, 0xe, 0x10100c0000eb9b0, 0x0)
        /home/itaysk/.asdf/installs/golang/1.13.6/packages/pkg/mod/github.com/iovisor/[email protected]/bcc/module.go:202 +0x26
github.com/iovisor/gobpf/bcc.(*Module).LoadKprobe(...)
        /home/itaysk/.asdf/installs/golang/1.13.6/packages/pkg/mod/github.com/iovisor/[email protected]/bcc/module.go:177
github.com/aquasecurity/tracee/tracee.(*Tracee).initBPF(0xc0000ba280, 0xc00012c000, 0xad01, 0x0, 0x61e3c1)
        /home/itaysk/dev/tracee/tracee/tracee.go:237 +0x579
github.com/aquasecurity/tracee/tracee.New(0xc000070800, 0x3f, 0x3f, 0x0, 0x61e3c1, 0x5, 0x40, 0xc0000b86c0, 0x0, 0x0)
        /home/itaysk/dev/tracee/tracee/tracee.go:193 +0x1cd
main.main.func1(0xc0000b8680, 0xc000080b00, 0xe)
        /home/itaysk/dev/tracee/main.go:33 +0x2fc
github.com/urfave/cli/v2.(*App).RunContext(0xc000066600, 0x666060, 0xc00008a020, 0xc0000721b0, 0x1, 0x1, 0x0, 0x0)
        /home/itaysk/.asdf/installs/golang/1.13.6/packages/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311 +0x6be
github.com/urfave/cli/v2.(*App).Run(...)
        /home/itaysk/.asdf/installs/golang/1.13.6/packages/pkg/mod/github.com/urfave/cli/[email protected]/app.go:211
main.main()
        /home/itaysk/dev/tracee/main.go:84 +0x670

fix timestamp resolution in go

should be the same as python

Support all x86-64 syscalls

We currently support ~80 x86-64 system calls.
Add support for the remaining ones.

BUG: go version of tracee always reports 0 lost events

In the statsStore there is a lostEvCounter, but it is never altered.
The code around https://github.com/aquasecurity/tracee/blob/master/tracee/tracee.go?#L432

		case lost := <-t.lostEvChannel:
			t.stats.errorCounter.Increment(int(lost))
		}
	}

should probably be incrementing this counter rather than the general errorCounter.

executed files are sometimes missed

(reported by Idan)
probably because the container exists quickly

Capture exec not working for containers

Capture exec uses the absolute path in the mount namespace of the container.
This path is not accessible directly from the host mount namespace.
We need to use setns or copy from /proc/pid/root in such cases.

show-all-syscalls on by default in go

should be opt in

what is the difference compared with falco?

release tracee

build and release artifacts for tracee. probably with goreleaser. will need to version before.
@lizrice what do you think about the version:

1. officially "start counting" - 0.0.1
2. don't start counting 0.0.0
3. 1 with exnension: 0.0.1-rc1
4. 2 with extension: 0.0.0-rc1
   Or if you have another suggestion. Note that this will be the git tag so applied to python as well

Can't trace some system call in multi-thread program

Hi. I'm studying on research work about system call tracing through eBPF and Tracee is the most suitable tool for my project.
But I have no idea about the reason why this problem happen.

I wrote a simple C code(TargetApp.c) below to see if Tracee can trace all syscalls in the system.
In this code, 16 child processes are forked and each child process executes getpid 62,500 times.

int numOfCallsInLoop = 1000000;    // 1 million
int num_proc = 16;
int i, j, pid;

for(i=0; i<num_proc; ++i){
    pid=fork();
    if(pid == 0){
        for(j=0; j<numOfCallsInLoop/num_proc; ++j){
            syscall(39);    // getpid
        }
        return 0;
    }else if(pid == -1){
        return 1;
    }
}
for(i=0; i<num_proc; ++i){
    wait(NULL);
}

Therefore, the total number of getpid syscall is 1 million and I expected that sum of eventCounter and LostEvCounter should be more than 1 million.
However, the number of events shown in Tracee was below, sum of those two value was only 387,569 (=307,235+80,334).
So it was too less than I expected.

Test Environment
Num of Core: 8
OS: Ubuntu 20.04 server
Kernel: 5.4.0-45-generic

Tracee start command: sudo ./tracee -e getpid
(By the way, Tracee don't show any event when I run that TargetApp code after the commit "Handle events parameters types and names using parameters map [915a1cc]". So I used the previous version)

Integrating tracee in Yocto build

Please guide me with the steps to integrate tracee in yocto build

Support 32bit executables

Currently, tracee doesn't trace system calls of 32bit binaries (that run on x64).
This may be fixed by changing syscalls attach mechanism to tracepoints instead of kprobes

Timestamp resolution in go version should be microseconds

Unlike the python version, the timestamp in the go version is shown in seconds resolution.
This resolution is too coarse and needs to be changed to microseconds resolution

improve capture executed files

From: #208

there's still an edge case where capture executed files won't work: if a container with shared pidns started and exists, and another one is started that uses the same mntns, then the pids cache won't get updated. It's still an improvement over the current state, but not perfect.

to fix that we need a way to know when a mnt ns is created (or destroyed), and clear the mntns cache record

related #215

Additional tracing modes

We currently support three modes of operation. It may be useful to add three more.
As we don't want to have a flag for each of these modes, we better change the UX to use "--trace-target" as suggested below by @itaysk

trace target	trace mode	status (issue)
process	all	TODO
process	new	implemented, migrate UX
process	specific	implemented, migrate UX
container	all	TODO
container	new	implemented, migrate UX
container	specific	TODO #255

Custom output format using gotemplate

Allow consumers of tracee to customize the output format by providing a gotemplate file via the output flag:
tracee -o gotemplate=/path/to/template/file

some question about execve problem bcc #2627

I tested about execve problem(bcc #2627 test program) in my ubuntu.
I tested it in kernel version 4.18 and 5.3.
But tracee recorded execve's parameter ('/bin/ls') well.
I can't understand why tracee records parameters well in my ubuntu...

Does any patch applied?
And is this problem related with bpf_probe_read() exactly?

another question... about secure tracing section in README...

When Tracee reads information from user programs it is subject to a race condition where the user program might be able to change the arguments after Tracee has read them.

This line just written from guessing which based on execve problem(bcc #2627), isn't it?

Merge arg names and values arrays into a map from name to value

#165 added argument names to the output. Internally this is managed as 2 arrays once for names and another for values, with index shared between them. This is awkward for external usage, so when building the exported Event we should make this into a map, from name to value.

There's a potential issue if the map does not maintain the order of elements, need to investigate this.

mentioned in review: #165 (comment)

possibly incorrect mode_t size

noticed that we read mode_t as uint32 but it might be unsigned short? (architecture dependent?)

tracee/tracee/tracee.go

Line 904 in de74185

mode, err := readUInt32FromBuff(dataBuff)

https://elixir.bootlin.com/linux/latest/source/include/linux/types.h#L18

tracee in Kubernetes

tracee should be able to run in a Kubernetes. the goal is for a user to be able to start tracing one pod or container, without jumping through too many hoops.
this issue is for tracking Kubernetes related requests and tasks.

To be discussed:

• running tracee in pods vs running on nodes.
• what are the implications of running many instances of tracee on the same node?
• client/server architecture can be used to separate event collecting from filtering.
• we currently do detect containers, but we don't allow to target a specific container.
• consider filtering by mnt ns and cgroups

related to #20

tracee in container

tracee should be able to run in a container. this issue is for tracking container related requests and tasks.

Additional filtering

I'd like to be able to filter by UTS name field.

Option 1: add --u <uts_name>[,<uts_name2>...] / --uts-name <uts_name1>[,<uts_name2>...] option which would only output a line if it matches one of the specified UTS names

Option 2: add -f <field name>=<value1>[,<value2>...] / --filter <field name>=<value>[,<value2>...] which would only output a line if the field matches the specified values. If multiple filters are specified, they are ANDed together (so the line only gets matched if all filters have a match).

Option 1 is easier to both implement and use, and we already have --event for filtering on particular events, so I'm leaning towards that. It wouldn't rule out option 2 for the future. Wdyt?

Convert BCC to libbpf+CO-RE

All the reasons why and a practical guide for how to do that: https://facebookmicrosites.github.io/bpf/blog/2020/02/20/bcc-to-libbpf-howto-guide.html

Work in progress in the branch libbpf https://github.com/aquasecurity/tracee/tree/libbpf

Error running Tracee when compiled from source

Hi,

I'm building tracee on Ubuntu 18.04. I have built and installed bcc from source as mentioned in the following instructions: https://github.com/iovisor/bcc/blob/master/INSTALL.md#ubuntu---source

I am able to build tracee successfully, but when I run it I get the following error:

vagrant@vagrant:~/dev/tracee$ dist/tracee
/virtual/main.c:1284:1: warning: declaration of 'struct tracepoint__raw_syscalls__sys_enter' will not be visible outside of this function [-Wvisibility]
TRACEPOINT_PROBE(raw_syscalls, sys_enter) {
^
/virtual/include/bcc/helpers.h:996:46: note: expanded from macro 'TRACEPOINT_PROBE'
int tracepoint__##category##__##event(struct tracepoint__##category##__##event *args)
                                             ^
<scratch space>:151:1: note: expanded from here
tracepoint__raw_syscalls__sys_enter
^
/virtual/main.c:1300:47: error: incomplete definition of type 'struct tracepoint__raw_syscalls__sys_enter'
    save_to_submit_buf(submit_p, (void*)&(args->id), sizeof(int), INT_T);
                                          ~~~~^
/virtual/main.c:1284:1: note: forward declaration of 'struct tracepoint__raw_syscalls__sys_enter'
TRACEPOINT_PROBE(raw_syscalls, sys_enter) {
^
/virtual/include/bcc/helpers.h:996:46: note: expanded from macro 'TRACEPOINT_PROBE'
int tracepoint__##category##__##event(struct tracepoint__##category##__##event *args)
                                             ^
<scratch space>:151:1: note: expanded from here
tracepoint__raw_syscalls__sys_enter
^
1 warning and 1 error generated.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x533556]

goroutine 1 [running]:
github.com/iovisor/gobpf/bcc.(*Module).Load(0x0, 0xc00001ac40, 0xe, 0x2, 0x0, 0x0, 0xe, 0x80, 0x0)
	/go/pkg/mod/github.com/iovisor/[email protected]/bcc/module.go:202 +0x26
github.com/iovisor/gobpf/bcc.(*Module).LoadKprobe(...)
	/go/pkg/mod/github.com/iovisor/[email protected]/bcc/module.go:177
github.com/aquasecurity/tracee/tracee.(*Tracee).initBPF(0xc0000b6300, 0xc000124000, 0xb350, 0x0, 0x61f4a2)
	/tracee/tracee/tracee.go:235 +0x58f
github.com/aquasecurity/tracee/tracee.New(0xc00000ad00, 0x40, 0x40, 0x0, 0x61f4a2, 0x5, 0x40, 0xc00005a7c0, 0x0, 0x0)
	/tracee/tracee/tracee.go:192 +0x1cd
main.main.func1(0xc00005a780, 0xc00000ec00, 0x10)
	/tracee/main.go:36 +0x351
github.com/urfave/cli/v2.(*App).RunContext(0xc000001b00, 0x667440, 0xc00001a0c0, 0xc0000121e0, 0x1, 0x1, 0x0, 0x0)
	/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311 +0x6be
github.com/urfave/cli/v2.(*App).Run(...)
	/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:211
main.main()
	/tracee/main.go:93 +0x74d

Any assistance would be greatly appreciated.

Badges in README.md file

Badges help in increasing the readability of the readme file and readers get a clear idea of the repository very quickly by scanning the attached badges.

We can add below badges:

• Github release
• Go Report Card
• GitHub License
• Docker pulls count

Don't print raw_syscalls for syscalls with existing events

Currently we have duplication of syscall events where raw_syscalls prints an event in addition to an existing syscall event.
Remove the raw_syscalls print in that case

Hacktoberfest at Aqua Security

Dear community, we appreciate you and your contribution to the success of our open source projects! In celebration of Hacktoberfest, we would like to extend our appreciation and reward your hard work. This repository is participating in Aqua Security 2020 Hacktoberfest campaign. For full details and guidelines please visit the official repo: Hacktoberfest at Aqua Security.

New bpf features to consider

As bpf evolves, it gets new features in almost every new kernel release.
Following is a list of features to consider integrating with tracee:

1. kernel 4.16: functions can be declared without “always inline”. This may reduce instructions count and program size
2. kernel 4.18: Use cgroup id for containers (bpf_get_current_cgroup_id()). Another bpf helper allows getting the cgroup ids of an ancestor at some level (can be used to set policies for container ids)
3. kernel 5.3: bpf bounded loops - https://lwn.net/Articles/794934/
4. kernel 5.5: Use fentry fexit program types (based on bpf trampolines) which have zero overhead compared to kprobes. https://lwn.net/Articles/804937/
5. kernel 5.8: New capabilities - cap_bpf, cap_perfmon - https://lwn.net/Articles/822362/
6. kernel 5.8: the preferred way to submit events is using “ringbuf_output” - https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#4-bpf_ringbuf_output

index capture written files

add a file to the output directory that indexes the written files by their paths. for example:
/path/to/file -> write.dev-1.inode-33

Improve gobpf resiliency

With the rewrite from Python to Go, the are some issues that require investigation:

• Configurable perf map size
• Consider events channel size
• How to handle lost events
• Show events count and lost/err count in epilogue

missing output in docker?

Hi, I'm trying to deploy tracee as a docker container, I'm interested in extracting the output of tracee as a file.

According to the help manual

--output-path value                 set output path (default: "/tmp/tracee")

Expected behaviour:
The stdout should be located in /tmp/tracee

Actual behaviour:
no file is ever generated in /tmp/tracee

To reproduce:

# start container in detached mode
docker run -d --name tracee --rm --privileged -v /lib/modules/:/lib/modules/:ro -v /usr/src:/usr/src:ro aquasec/tracee:latest

# attach to container
docker exec -ti tracee bash

ls /tmp/tracee

I suspect I'm misunderstanding something because the tool is great!

Support vfs_writev

We currently support vfs_write,
however, when using writev and pwritev syscalls, the kernel uses the vfs_writev function, and not the vfs_write function.
Add support for this function as well.

Show syscall name in raw_syscall

Instead of showing the syscall number, we can print the syscall name

ARM support

Tracee currently doesn't work on ARM architecture. If you are interested in this, please add the 👍 reaction to this issue, and optionally add a comment to describe your use case.

From: #231 (reply in thread)

The reason why Tracee will not work on arm devices is that the event ids numbers match the x86_64 syscalls numbers.
I recently added a table that maps x86_64 syscalls to x86_32 syscalls to support 32bit applications running in compat mode.
We can do something similar for arm, but some other changes are also required, like reading the arguments to different registers in pt_regs struct in the bpf code.

Exit gracefully when not running as root

As described in #96, errors might occur when not running tracee as root.
We need to exit gracefully when this happens, and tell the user to run tracee as root

user customizable table layout

We currently support two "table" output formats - regular and verbose.
Let the user have the ability to choose how to format the table output -
which fields to include and their order

Arg names should not be parsed within the event processing loop

#165 added arg names which are encoded into a uint8, and decoded when reading the event fro the buffer. When reading the event, and also while processing it, theres no need for the arg name, the tag (uint8 encoding) suffice. the name decoding code should be moved out of the event loop and into a printing related region, presumably the newEvent function.

Mentioned in review: #165 (comment)

Build and push docker images to aquasec org

this will be a manual process for the time being

group similar syscalls in the ux

some sycalls have different variants: open/openat/open_by_handle_at, dup/dup2/dup3.
perhaps we can accept a flag -e open* -e dup* which will catch all related syscalls. we can do a literal search by prefix which will work for most examples.

raw_syscalls shouldn't be in the output by default

After merging 799ed4f, raw_syscalls events are now shown in the output even when not explicitly run with --raw-syscalls.
Need to disable these events by default, and only show them when --raw-syscalls flag is set

Add license

Merge Event IDs enum with EventsIDToName map

#168 introduced an enum which makes EventsIDToName redundant (or the other way around, although the former is preferable for performance). this is a reminder to merge the two.
Mentioned in review: #168 (comment)

Readme doc update

I would like to add the Table of Contents in the README.md file.
It has been added in a few other repos hence I believe it can be added in this repo as well.

@itaysk