elastic / makelogs

@elastic/makelogs

Simple generator used to push fake HTTP traffic logs into elasticsearch. It uses the node.js client.

This version of makelogs is designed to work with the latest builds from elasticsearch master, and is not compatible with previous versions.

Events are pushed into logstash-YYYY.MM.DD formatted indices (configurable using --indexPrefix), using the bulk API, and are easily consumed by Kibana.

Documents look like this:

{
  "_index": "logstash-2014.06.17",
  "_type": "_doc",
  "_id": "706786",
  "_score": 11.412156,
  "_source": {
     "index": "logstash-2014.06.17",
     "@timestamp": "2014-06-17T17:00:27.053Z",
     "ip": "225.27.202.82",
     "extension": "html",
     "response": "200",
     "geo": {
        "coordinates": [
           44.23107,
           -94.99893444
        ],
        "src": "IM",
        "dest": "PK",
        "srcdest": "IM:PK"
     },
     "@tags": [
        "error",
        "info"
     ],
     "utc_time": "2014-06-17T17:00:27.053Z",
     "referer": "http://nytimes.com/error/gemini-11",
     "agent": "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1",
     "clientip": "225.27.202.82",
     "bytes": 5108.1583889899775,
     "request": "/ivan-bella.html",
     "@message": "225.27.202.82 - - [2014-06-17T17:00:27.053Z] \"GET /ivan-bella.html HTTP/1.1\" 200 5108.1583889899775 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1\"",
     "spaces": "this   is   a   thing    with lots of     spaces       wwwwoooooo",
     "xss": "<script>console.log(\"xss\")</script>",
     "headings": [
        "<h3>robert-satcher</h5>",
        "http://twitter.com/success/scott-altman"
     ],
     "links": [
        "[email protected]",
        "http://www.slate.com/info/boris-yegorov",
        "www.twitter.com"
     ],
     "machine": {
        "os": "win 7",
        "ram": 17179869184
     }
  }
}

to install

npm install -g @elastic/makelogs

then run makelogs --help for usage info:

$ makelogs --help
A utility to generate sample log data.

Usage: makelogs [options]

Options:
  ...

The tool is, admittedly, not super configurable. Just tell it how many events you want, how many days to generate data for, and it will cruise.

Do not use this on any sort of production elasticsearch installation.

The event stream can be a tad unforgiving, and could cause some damage to an elasticsearch cluster under load. It is designed for debugging locally.

makelogs's Issues

Revert 86537a

We need to revert jbudz@86537a7 once elastic/elasticsearch#13112 is closed

Single type

Types are being removed, see elastic/elasticsearch#24428. We need to collapse the nginx/apache types.

Makelogs doesn't work without --host set from the CLI

When I run makelogs without --host localhost:9200 set it throws an error when splitting by :

Slow log creation

$ time makelogs -c 300k -d 5
Generating 300000 events from 2015-06-19T00:00:00+00:00 to 2015-06-29T23:59:59+00:00
......................................................................................................

created 300000 events

real    16m19.699s

This is when running ES in a constrained vm, so the slowness it really amplified. In version 1.4.1 and previous, that would take a minute or 2 at most.

RandomList function return undefined

Hi,
In makelogs\samples\random_list.js file you have RandomList function, this function return sometimes undefined, for example in case the list length is 2, and Math.random() return 0.9, so the function return list[2], there is nothing in this place, I think you need to replace Math.round with Math.floor

Add ability to introduce fake anomalies.

I would love to be able to have a way to say, generate 10 days worth of data but add spikes for different countries, or add more 500/404 response codes at random intervals etc. Does that make sense?

Exception when specifying custom date range

When specifying a custom date range using --days switch I hit an exception:

$ makelogs --days "-10,+10"

  throw new TypeError('Unable to determine the starting and end dates.');
        ^
TypeError: Unable to determine the starting and end dates.

When digging into the code I noticed that the above is translated by optimist into:

{
  ...
  days: true,
  d: true
  ...
}

I tried several different conventions for specifying lower/upper bound, like 10,10, -10,10, 10,+10 and all resulted with the above exception.

Is there a reason for requiring the -days,+days format or can we assume the first element will always we the lower bound to subtract from the base date?

Here's a small change for your review:
itayw@522e605

Thanks for this great tool!

Update lodash

The lodash dependency is quite out of date, and is commonly flagged by security scanners. It'd be great to update this to a modern version of lodash.

Incorrect detection of existing index templates

When I start makelogs on a totally fresh instance of Elasticsearch, I get the message about existing index templates, asking me if they can be replaced:

? Existing logstash-* indices and/or index templates were found, can they be replaced?

Odd, since this is a new ES instance. When I select yes, ES shows an error:

[2017-07-24T11:27:16,082][DEBUG][o.e.a.a.i.t.d.TransportDeleteIndexTemplateAction] [IWB2yw7] failed to delete templates [makelogs_index_template__logstash-]
org.elasticsearch.indices.IndexTemplateMissingException: index_template [makelogs_index_template__logstash-] missing

I think the detection of existing templates is broken. Observed against 5.5.0 in this case, but I've seen it happen in older versions as well.

Days flag seems wrong

$ date
Wed Sep 17 16:57:37 MST 2014
$ makelogs -c 150000 -d 0,3
Generating 150000 events from 2014-09-20T00:00:00+00:00 to 2014-09-17T23:59:59+00:00

Today is the 17th, but it doesn't start making logs until 1 seconds before the 18th. -d 0, to me, seems like it should start the logs today, at 2014-09-17T00:00:00+00:00. At the very least, it should start the logs at now(), no?

There is an error "Unable to load PFX certificate" when connecting the aws elasticsearch.

I tested with my ec2 and aws elasticsearch but got the errors.
How can I gernerate logs without an error below?

makelogs - url https://search-xxxxx-es-fd5uuxxxxx3dvjjshurlfuwn4fq.ap-northeast-2.es.amazonaws.com/

node:internal/tls/secure-context:278
context.loadPKCS12(toBuf(pfx));
^

Error: Unable to load PFX certificate
at configSecureContext (node:internal/tls/secure-context:278:15)
at Object.createSecureContext (node:_tls_common:116:3)
at Object.connect (node:_tls_wrap:1613:48)
at HttpsAgent.createConnection (node:https:143:22)
at HttpsAgent.createSocket (/home/ec2-user/.nvm/versions/node/v16.5.0/lib/node_modules/@elastic/makelogs/node_modules/agentkeepalive/lib/_http_agent.js:265:26)
at HttpsAgent.createSocket (/home/ec2-user/.nvm/versions/node/v16.5.0/lib/node_modules/@elastic/makelogs/node_modules/agentkeepalive/lib/agent.js:77:11)
at HttpsAgent.addRequest (/home/ec2-user/.nvm/versions/node/v16.5.0/lib/node_modules/@elastic/makelogs/node_modules/agentkeepalive/lib/_http_agent.js:239:10)
at new ClientRequest (node:_http_client:305:16)
at Object.request (node:https:353:10)
at HttpConnector.request (/home/ec2-user/.nvm/versions/node/v16.5.0/lib/node_modules/@elastic/makelogs/node_modules/elasticsearch/src/lib/connectors/http.js:182:23) {
code: 'ERR_CRYPTO_OPERATION_FAILED'
}

Omit option doesn't remove field from mappings

When using the omit option the fields still get sent to the index via the field mappings.

Makelogs should install the logstash dynamic template before it starts indexing

The logstash dynamic template for Elasticsearch is very useful and it would be great if makelogs could check for that and if its not there, install it. Right now I need to add it to get the Raw fields.

Support for datastreams

🔖 Feature description

There is only a feature to give indexPrefix in the command line arguments and not an index or datastream itself.

🎤 Why is this feature needed ?

Given the advent of datastreams in the Elastic community, this feature would help folks generate logs and ingest it in a datastream post which they can test the behaviour of datastreams in various scenarios.

✌️ How do you aim to achieve this?

I am really not sure what I can do from my end. Maybe just allow an optional argument to pass explicit indexName or datastream

🔄️ Additional Information

Not applicable

👀 Have you spent some time to check if this feature request has been raised before?

I checked and didn't find similar issue

Are you willing to submit PR?

Not really sure

Update schema and fields to match ECS

Personally, I'd love to see makelogs modified to create logs in the format of ECS. Great for demos and we can adopt in sample data 😄

Add a little of reflection to push the correct index template depending on version

Query the cluster to understand the version used to push the correct index template.
Types between 2.x and 5 changed...

Observe the host's protocol

When parsing the host URL, the protocol is currently being ignored, so makelogs will only work with HTTP, since that's the default.

Compatibility with 5.x

There seems to be a compat issue with ES 5.x... I ran makelogs -c 10000 which created data in an index named logstash-0 and not time based ones (even though it's not a problem with kibana due to the field stats checking).

makelogs-3.0.2 creates single logstash-0 index

I believe previous versions of makelogs (2.x) created 3 time-based indices. Is this a regression in makelogs 3.x?

update-notifier dependency is out of date

If possible, the Kibana project would like to update the update-notifier dependency of this package to v6.0.2 or newer.

Bulk queue size is no longer dynamically updatable

makelogs sets the bulk queue size in Elasticsearch (threadpool.bulk.queue_size) to unlimited via the REST API. Starting with elastic/elasticsearch@da74323, this setting is no longer dynamically updatable. This is causing makelogs to fail with this error:

Error: [illegal_argument_exception] transient setting [threadpool.bulk.queue_size], not dynamically updateable
    at respond (/Users/shaunak/development/github/ycombinator/kibana/node_modules/makelogs/node_modules/elasticsearch/src/lib/transport.js:256:15)
    at checkRespForFailure (/Users/shaunak/development/github/ycombinator/kibana/node_modules/makelogs/node_modules/elasticsearch/src/lib/transport.js:219:7)
    at HttpConnector.<anonymous> (/Users/shaunak/development/github/ycombinator/kibana/node_modules/makelogs/node_modules/elasticsearch/src/lib/connectors/http.js:155:7)
    at IncomingMessage.wrapper (/Users/shaunak/development/github/ycombinator/kibana/node_modules/makelogs/node_modules/elasticsearch/node_modules/lodash/index.js:3095:19)
    at emitNone (events.js:72:20)
    at IncomingMessage.emit (events.js:166:7)
    at endReadableNT (_stream_readable.js:913:12)
    at nextTickCallbackWith2Args (node.js:442:9)
    at process._tickCallback (node.js:356:17)

Update inquirer

The inquirer dependency is quite out of date, and the transitive lodash dependency is commonly flagged by security vulnerability scanners.

Deprecate old published makelogs on npm

On NPM we have now the @elastic/makelogs and the previous makelogs with the last update to 4.3.0.
Could we in a way deprecate the makelogs published library and redirect to @elastic/makelogs for new updates?

TypeError: error.match is not a function

When runing makelogs with latest master of kibana (and es provided with npm run elasticsearch)
an exception is thrown

$ makelogs
Generating 14000 events from 2016-12-22T00:00:00Z to 2016-12-24T23:59:59Z
? Existing logstash-* indices and/or index templates were found, can they be replaced? Yes
clearing existing "logstash-*" index templates and indices
creating index template for "logstash-*"
TypeError: error.match is not a function
    at /home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/eventBuffer/_bulkQueue.js:45:23
    at Array.forEach (native)
    at /home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/eventBuffer/_bulkQueue.js:37:20
    at tryCatcher (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/promise.js:510:31)
    at Promise._settlePromiseAt (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/promise.js:584:18)
    at Async._drainQueue (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/async.js:128:12)
    at Async._drainQueues (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:574:20)
    at tryOnImmediate (timers.js:554:5)
    at processImmediate [as _immediateCallback] (timers.js:533:5)
undefined
TypeError: error.match is not a function
    at /home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/eventBuffer/_bulkQueue.js:45:23
    at Array.forEach (native)
    at /home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/eventBuffer/_bulkQueue.js:37:20
    at tryCatcher (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/promise.js:510:31)
    at Promise._settlePromiseAt (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/promise.js:584:18)
    at Async._drainQueue (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/async.js:128:12)
    at Async._drainQueues (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/home/ppisljar/.nvm/versions/node/v6.4.0/lib/node_modules/makelogs/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:574:20)
    at tryOnImmediate (timers.js:554:5)
    at processImmediate [as _immediateCallback] (timers.js:533:5)

Ability to pass in requestTimeout

When trying to use makelogs on remote Elasticsearch instances, the default 30s requestTimeout value is often not enough. The ability to pass in your own setting would be really helpful.