javiroman / flume-snmp-source

Apache Flume Source plugin for SNMP Query and Trap consumption.

My Informal Profile	My Blog	My Hobby
Technical CV	Big Data Blog	Street Photography

---

Repositories Summary

Working Areas	Repository	Description
Kubernetes Big Data Engineering Group	kubernetesbigdataeg	Community resource for big data and data science related software projects
AntelopeDB Project	antelopedb	AntelopeDB HTAP Project fork for contributing
Apache NiFi	nifi	Apache NiFi fork for contributing
	nifi-prometheus-bundle	Apache NiFi Prometheus Remote Write Processor
Fedora Packages	mesos-rpm	Apache Mesos Fedora Package
	zookeeper-rpm	Apache Zookeeper Fedora RPM Package
	yetus-rpm	Apache Yetus Fedora RPM Package
Testbed Environments	docker-hadoop	Docker Compose definitions for Big Data Testing
	dfsadmin-inotify	Hadoop DFSAdmin administrative API Java test
	disk-stress	Python Based Disk Stress tool for Kubernetes/Mesos limits testing
	system-stress	ANSI C based Host System Stress Tool
	k8shelloworld	Small Tools for exploring Kubernetes capabilities
	TestBeds	Vagrant environments for technology testing
Misc	patternfly-react-seed-login	React PatternFly Login Example
	hpcservergridscheduler	HPC Grid Scheduler for Octave/Matlab based on ICE RPC Framework
	gridexecutor	HPC Grid Scheduler for Massive Remotely Command Execution based on ICE RPC Framework
	rlink	Makefile remote linking tool based on ICE RPC Framework
Java PoCs	jremotewrite	Java Prometheus Remote Write Endpoint
	jcsfcache	COW Caching Research based on Apache Commons Java Caching System (JCS)
	jschexecute	JSch Remote Execution Research
	jreactapp	Jetty, Jersey and ReactJS Spike
	jrtsp-client	Java RTSP Client based on Netty NIO Framework

flume-snmp-source's Issues

SimpleDateFormat("MM dd HH:mm:ss yyyy") is not used

The line:

DateFormat dateFormat = new SimpleDateFormat("MM dd HH:mm:ss yyyy");

is not used in the code. Must be dropped.

agent.sources.source1.host - list of hosts or file

It would be interested pass a list of hosts or file with a list of hosts in order to send the SNMP GETBULK to this set, better than repeat a lot of sources in the configuration file (we are talking aboug thousand of hosts to query).

OID non exits error not catched

Whe the OID is not available in the managed system:

snmpbulkwalk -v2c -Cn0 -Cr1 -c public 22.79.52.119 1.3.6.1.4.1.2000.1.2.5.1.22
iso.3.6.1.4.1.2000.1.2.5.1.22 = No Such Object available on this agent at this OID

However the code is inserting a number "1" in the erroneous OID field.

Make parsing Trap PDU as optional

Can make parsing Trap PDU as optional. Flume just need to carry the data as byte[], hence parsing here may not provide a lot of benefit. This may not be possible with snmp4j, but using raw (N)IO library like Apache MINA or Netty. I wrote something similar, which can be reused here.

There may be situations where parsing a PDU might be of help. My knowledge is a bit rusty, but do we receive Traps on TCP? I remember only UDP. If so, the we don't need to add TCP mapping.

log INFO everywhere .... we have to tune this.

The message in SNMPQuery is logged in the flume.log file.

When query or trap fails send message to channel incomplete

For example when the query fails (timeout ... whatever) the message to pass toward the Channel is build with incomplete data: Thu Aug 28 18:48:11 CEST 2014,23.23.52.11 ..... only the Date field and the host IP field.