akagi201 / jsonbeat

Jsonbeat is a JSON log harvester, mostly used to fetch logs files and feed them into Elasticsearch.

Jsonbeat

Welcome to Jsonbeat.

Ensure that this folder is at the following location: ${GOPATH}/src/github.com/Akagi201/jsonbeat

Getting Started with Jsonbeat

Requirements

• Golang 1.7

Init Project

To get running with Jsonbeat and also install the dependencies, run the following command:

make setup

It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.

To push Jsonbeat in the git repository, run the following commands:

git remote set-url origin https://github.com/Akagi201/jsonbeat
git push origin master

For further development, check out the beat developer guide.

Build

To build the binary for Jsonbeat run the command below. This will generate a binary in the same directory with the name jsonbeat.

make

Run

To run Jsonbeat with debugging output enabled, run:

./jsonbeat -c jsonbeat.yml -e -d "*"

Test

To test Jsonbeat, run the following command:

make testsuite

alternatively:

make unit-tests
make system-tests
make integration-tests
make coverage-report

The test coverage is reported in the folder ./build/coverage/

Update

Each beat has a template for the mapping in elasticsearch and a documentation for the fields which is automatically generated based on fields.yml by running the following command.

make update

Cleanup

To clean Jsonbeat source code, run the following commands:

make fmt
make simplify

To clean up the build directory and generated artifacts, run:

make clean

Clone

To clone Jsonbeat from the git repository, run the following commands:

mkdir -p ${GOPATH}/src/github.com/Akagi201/jsonbeat
git clone https://github.com/Akagi201/jsonbeat ${GOPATH}/src/github.com/Akagi201/jsonbeat

For further development, check out the beat developer guide.

Packaging

The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:

make package

This will fetch and create all images required for the build process. The hole process to finish can take several minutes.

jsonbeat's Issues

Doesn't respond to Ctrl+C when run in debug mode

2018/09/26 14:58:58.690603 metrics.go:39: INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=4473924 beat.memstats.memory_alloc=1102512 beat.memstats.memory_total=1102512
^C2018/09/26 14:59:04.801659 service.go:33: DBG Received sigterm/sigint, stopping
^C^C^C^C^

Requires "kill -9".

This situation is when it has processed all lines in the file. If you add another line after pressing Ctrl+C then it stops! Looks like a go routine/channel issue.

This is on Mac 10.13.6, go 1.11