micrictor / enrichserver

Modular data enrichment HTTP API

EnrichServer

An asynchronous, modular HTTP server for enrichment of arbitrary data.

Written in Python3, supports HTTP GET/POST requests and in-memory caching of results.

Usage

To start the server, execute EnrichServer.py. The server listens on port 8080 by default

API requests roughly follow the format of http://<server>/api/<endpoint>?<param1>=<val1>&<param2>=<val2>...

The following API endpoints are available.

• api/<module>?info - Returns all available metadata on the module
• api/<module>?<params>

If debug is enabled with the --debug flag:

• debug/list - Returns a list of the currently available modules
• debug/cache - Returns the current cache of results

POST requests may be made to the same endpoints, with arguments passed in key-value pairs or as JSON.

JSON body should be in the following format

{
	"action":<enrich|info>,
	"module":<module_name>,
	"args": {"data":"val", "data2","val2", ...}
}

All modules must be stored in ./modules/, relative to pwd of EnrichServer.py.

Modules must implement a class named Enricher as a subset of the BaseEnricher.BaseEnricher class.

Modules should include an override for the do_enrich method, where the arguments to the enricher are passed as a dict.

Modules may enable in-memory caching of results by setting the member variable shouldCache to True.

NOTE: Caching is limited to a default of 100 entries per module by default

Currently provided modules are as follows:

• Sleeper - Sleeps for 10 seconds then echos the data parameter; Used to test asynch
• StrRev - Returns the mirror of the data parameter
• Entropy - Returns the Shannon entropy of the data parameter over an optional charset parameter( Default to DNS ASCII charset )

Planned modules:

• RDAP lookup of IP/domain
• WHOIS - Return WHOIS data for domain provided in data
• ASN - Provide ASN information(including owner) of IP address in data
• ES_Query - Return the result of query against es_server