sematext / logsene-js

Log shipping library for sending logs to Sematext from JavaScript apps

JavaScript client for Sematext Logs.

Installation

npm install logsene-js --save

Usage

var Logsene = require('logsene-js')
var logger =  new Logsene ('LOGS-APP-TOKEN')
logger.log ('info', 'text message', {tags: ['a','b'], customField: 'custom-field'})

Constructor Parameters

• token - Logs App token. Sign up here.
• type - Optional. Default type of your logs - please note you can define Elasticsearch mapping templates in Sematext Logs
• url - Sematext Logs receiver URL (e.g. if you are shipping logs to Sematext Enterprise), defaults to 'https://logsene-receiver.sematext.com/_bulk'
• storageDirectory - Directory where to buffer logs in the case of failure
• options:
  • useIndexInBulkUrl - If set to 'false' /_bulk will be used /indexName/_bulk otherwise.
  • httpOptions - general HTTP/HTTPS options for the request, e.g. SSL key,cert,passphrase,ca,rejectUNauthorized etc.
  • silent - If set to true, logsene-js will not log debug and errors to stdout. Used for prod envs, but not mandatory. This is false by default.

Special fields for indexing

In general Elasticsearch > 2.3 (including Sematext Logs) does not allow fields with leading underscore or dots in field names. Logsene-js converts such fields names (e.g. removing leading underscores, and replaces dots to underscores). However, a few fields are interpreted for indexing before renaming the fields:

• _type - used as '_type' in the index operation (bulk indexing)
• _id - used as '_id' in the index operation (bulk indexing)

Environment variables

• LOGS_TMP_DIR - Directory to store failed bulk requests, for later re-transmission. Failed requests are not stored, when LOGS_TMP_DIR is not set.
• LOG_INTERVAL - Time to collect logs before a bulk request is done. Default 10000 ms
• LOGS_BULK_SIZE - Maximum size of a bulk request. Default 1000.
• LOGS_MAX_MESSAGE_FIELD_SIZE - maximum size of the 'message' field, default 240k
• LOGS_RECEIVER_URL - URL for the Sematext Logs receiver. E.g. for Sematext Enterprise version. Defaults to Sematext Logs SaaS receiver https://logsene-receiver.sematext.com/_bulk
• LOGSENE_TMP_DIR - Directory where to buffer logs if the Elasticsearch endpoint is unavailable. If this is not set, logsene-js will not buffer logs at all.
• LOGS_BUFFER_ON_APP_LIMIT - default 'true'. HTTP bulk requests are retried until Sematext starts accepting logs again. Please increase your daily limit setting in Logs App settings if you see "403, App limit reached" errors. Setting the value to 'false' will disable disk buffering only for failed http requests '403, Forbidden / App limit reached'. We recommend to keep the default setting 'true' to avoid any loss of logs.
• LOGS_REMOVE_FIELDS - a comma separated list of fields, which should not be logged. For nested objects use a dot notation e.g. 'reques.body,request.size'

Note: Previous versions of logsene-js used LOGSENE instead of LOGS prefix for the settings above. Logsene-js is backward compatible to previous environment variable names. However all variable names with the LOGSENE prefix are depracated and might be removed in future relases.

Security

• HTTPS is enabled by default
• Environment variables for Proxy servers:
  • For HTTPS endpoints (default): HTTPS_PROXY / https_proxy

        export HTTPS_PROXY=https://my-ssl-proxy.example
        export HTTPS_PROXY=http://my-proxy.example

• For HTTP endpoints (e.g. On-Premises): HTTP_PROXY / http_proxy

        export HTTP_PROXY=http://my-proxy.example
        export HTTP_PROXY=https://my-ssl-proxy.example

Other related modules

• Please check winston-logsene a transport layer for the winston logging framework.
• Please see bunyan-logsene a transport layer for the bunyan logging framework.

logsene-js's Issues

If the sematext api returns a 400 (missing API token, for example), the library throws an error

This was the root cause which prompted me to consider #22

The library attempts to parse the body of every response from sematext without first checking that the overall request was successful.

The issue is on this line: https://github.com/sematext/logsene-js/blob/master/index.js#L528

The library should first check that the response is within the 200 > 299 range, and perhaps even then, safely check if res.body.items exists before trying to iterate it.

Additionally, mutating the response object here https://github.com/sematext/logsene-js/blob/master/index.js#L515 is not a great idea and should probably be assigned to a new variable.

I'll fix this as part of the work for #22

Missing module 'fs'

I get the following error during compilation, am I missing anything?

Module not found: Can't resolve 'fs' in '/home/michal/ev/eai-platform/node_modules/logsene-js'

dependencies from my package.json:

"dependencies": {
    "@apollo/react-common": "3.1.4",
    "@apollo/react-components": "^3.1.5",
    "@apollo/react-hooks": "3.1.5",
    "@apollo/react-ssr": "^4.0.0",
    "@types/micro-cors": "^0.1.1",
    "@types/mongodb": "^3.5.25",
    "@types/react-scroll": "^1.5.5",
    "apollo-boost": "^0.4.9",
    "apollo-cache": "^1.3.5",
    "apollo-cache-inmemory": "^1.6.6",
    "apollo-client": "^2.6.10",
    "apollo-link": "1.2.14",
    "apollo-link-http": "1.5.17",
    "apollo-link-schema": "1.2.5",
    "apollo-server-micro": "^2.14.2",
    "apollo-utilities": "^1.3.3",
    "bootstrap": "^4.5.0",
    "date-fns": "^2.15.0",
    "fs": "^0.0.1-security",
    "graphql": "^14.6.0",
    "graphql-tag": "^2.10.3",
    "gray-matter": "^4.0.2",
    "logsene-js": "^1.1.72",
    "micro": "^9.3.4",
    "micro-cors": "^0.1.1",
    "mongodb": "^3.5.9",
    "next": "^9.4.4",
    "next-with-apollo": "^5.1.0",
    "react": "^16.13.1",
    "react-bootstrap": "^1.2.2",
    "react-cookie-consent": "^5.1.3",
    "react-dom": "^16.13.1",
    "react-scroll": "^1.8.1",
    "remark": "^12.0.1",
    "remark-html": "^11.0.2"
  },

Huge npm package size

It seems like the YOUR_TOKEN_SIZE folder has been included in the bundle for some reason, resulting in package sizes of 195 MB.

Index conditional on message

Not sure this is applicable for the logsene use case, but for indexing to private elasticsearch instances, it would be nice to allow the index name to be determined by data from the message. The typical case would be for messages to be indexed into the appropriate time based index (eg foo-YYYY.MM.DD) based on the message timestamp.

Is that something you would be willing to support? Ideally, I'd like to be able to define the index as a function to allow maximum flexibility. Maybe something like this around here?

var _index = this.token;
if (typeof(_index) === 'function') {
  _index = _index(msg);
}

Module require incompatibility of node-fetch starting in 1.1.76

During a recently update from 1.1.75 -> 1.1.76, my service started erroring at runtime with this message:

TypeError: fetch is not a function
    at Logsene.send (/path/to/build/service.js:112653:3)
    at Timeout._onTimeout (/path/to/build/service.js:112351:12)
    at listOnTimeout (node:internal/timers:564:17)
    at process.processTimers (node:internal/timers:507:7)

This build uses typescript and webpack. I poked around the transpiled output, console logged fetch, and found that the imported value of fetch from node-fetch was actually a module, not a fetch function. It looked something like: {FetchErrors, Headers, Request, Response, default}. So it seems somewhere there's a mismatch about the style of module (commonjs, esm, etc).

If it's of interest the transpiled line looked like this:

const fetch = __webpack_require__(/*! node-fetch */ "./node_modules/node-fetch/lib/index.mjs")

I'm not the most well versed in the different types of module bundling so I'm not sure what the fix here is, or if it's an issue in my build or in logsene-js configs. I should also note I have another library gaxios that also depends on node-fetch and it does not have any issues. The difference seems to be that gaxios has compiled lib files bundled with it.

Thanks for your help.

The 'Request' library is deprecated

See https://github.com/request/request and request/request#3142

This library should probably be using node-fetch

throws error and causes app to crash when sematext server is down

Error [ERR_UNHANDLED_ERROR]: Unhandled error. ({
  source: 'logsene-js',
  err: {
    message: 'HTTP status code:503',
    httpStatus: 503,
    httpBody: '<html>\r\n' +
      '<head><title>503 Service Temporarily Unavailable</title></head>\r\n' +
      '<body>\r\n' +
      '<center><h1>503 Service Temporarily Unavailable</h1></center>\r\n' +
      '<hr><center>nginx/1.17.10</center>\r\n' +
      '</body>\r\n' +
      '</html>\r\n',
    url: 'https://logsene-receiver.sematext.com/_bulk'
  }
})
    at Logsene.emit (events.js:304:17)
    at Logsene.EventEmitter.emit (domain.js:485:12)
    at Request.httpResult [as _callback] (index.js:462:12)
    at Request.self.callback (node_modules/request/request.js:185:22)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:485:12)
    at Request.<anonymous> (node_modules/request/request.js:1154:10)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:485:12)
    at IncomingMessage.<anonymous> (node_modules/request/request.js:1076:12)

Add a constructor that does not require a type

A new Logsene user will not really know what the "type" parameter in the logsene-js ctor is all about - this type stuff is a notion from Elasticsearch that not everyone is familiar with. There should be a ctor that takes just the Logsene app token and uses a default Logsene type (I think the default type is "sematext")

Support AWS Elasticsearch

Supporting awsS4 signed HTTP requests would have the advantage that disk-buffer from logsene-js could be used with Amazon Elasticsearch. In addition, winston-logsene and Logagent could use Amazon Elasticsearch.

This npm module could make it easy to sign HTTP requests: https://www.npmjs.com/package/aws4

The current alternative is to use the Logagent AWS Elasticsearch plugin without disk-buffer.

On the other hand, logsene-js module is really made to work well with Sematext Cloud and local Elasticsearch servers and adding support for AWS Elasticsearch service might raise new issues like service specific error handling.

Errors in our builds

We are getting the following:
[14:15:28][Step 3/4] /opt/teamcity_agent/work/5b977a51079a5e19/node_modules/logsene-js/index.js:345
[14:15:28][Step 3/4] host_ip: ipAddress
[14:15:28][Step 3/4] ^^^^^^^
[14:15:28][Step 3/4]
[14:15:28][Step 3/4] SyntaxError: Unexpected identifier
[14:15:28][Step 3/4] at createScript (vm.js:80:10)
[14:15:28][Step 3/4] at Object.runInThisContext (vm.js:139:10)
[14:15:28][Step 3/4] at Module._compile (module.js:617:28)
[14:15:28][Step 3/4] at Object.Module._extensions..js (module.js:664:10)
[14:15:28][Step 3/4] at Module.load (module.js:566:32)
[14:15:28][Step 3/4] at tryModuleLoad (module.js:506:12)
[14:15:28][Step 3/4] at Function.Module._load (module.js:498:3)
[14:15:28][Step 3/4] at Module.require (module.js:597:17)
[14:15:28][Step 3/4] at require (internal/module.js:11:18)
[14:15:28][Step 3/4] at Object. (/opt/teamcity_agent/work/5b977a51079a5e19/node_modules/winston-logsene/lib/index.js:7:17)
[14:15:28][Step 3/4] npm ERR! Test failed. See above for more details.

Looks like there was a change in your code with no PR..

logsene-js/index.js

Lines 344 to 345 in e8ce6e4

	host: this.hostname
	host_ip: ipAddress

Missing a comma at line 344

"user strict" should be "use strict"

logsene-js/index.js

Line 18 in 1ccf7de

'user strict'

I'll fix this as a separate PR to keep them clean.

Inconsistent License notices

The license notification text in index.js is inconsistent with the Apache license in LICENSE

Support Elasticsearch 6.x API (no types)

Elasticsearch 6 can't process multiple log types in one index (breaking change). Logsene-js should support Elasticsearch 6 API.

Browser equivalent

This isn't really an issue, but I'm at a loss for where to look. I appreciate that many common use-cases demand these capabilities on the server, but there are definitely cases to be made for logging various things from the browser that RUM/Experience Monitoring just can't do (like being able to send logs from error boundaries in the React tree).

Is there a library supported by Sematext for browser logs? If not, can this library be migrated to be stream agnostic with stream adapters passed in at run time?

Node 18 compatibility

The lib does not declare compatibility with particular Node.js versions (e.g. via engines in package.json – maybe it should?), but it doesn't seem to work with Node 18:

logsene-js on  master [!] via ⬢ v18.1.0 is 📦 v1.1.75 
➜ npm -s run test
/home/…/logsene-js/node_modules/yargs/yargs.js:1172
      else throw err
           ^

TypeError: details.family.toLowerCase is not a function
    at /home/…/logsene-js/node_modules/ip/lib/ip.js:385:39
    at Array.filter (<anonymous>)
    at /home/…/logsene-js/node_modules/ip/lib/ip.js:384:37
    at Array.map (<anonymous>)
    at ip.address (/home/…/logsene-js/node_modules/ip/lib/ip.js:379:37)
    at Object.<anonymous> (/home/…/logsene-js/index.js:25:31)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Module._load (node:internal/modules/cjs/loader:827:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/home/…/logsene-js/test/test.js:3:15)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Module._load (node:internal/modules/cjs/loader:827:12)
    at Module.require (node:internal/modules/cjs/loader:1005:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at /home/…/logsene-js/node_modules/mocha/lib/mocha.js:334:36
    at Array.forEach (<anonymous>)
    at Mocha.loadFiles (/home/…/logsene-js/node_modules/mocha/lib/mocha.js:331:14)
    at Mocha.run (/home/…/logsene-js/node_modules/mocha/lib/mocha.js:809:10)
    at exports.singleRun (/home/…/logsene-js/node_modules/mocha/lib/cli/run-helpers.js:108:16)
    at exports.runMocha (/home/…/logsene-js/node_modules/mocha/lib/cli/run-helpers.js:142:13)
    at exports.handler (/home/…/logsene-js/node_modules/mocha/lib/cli/run.js:292:3)
    at Object.runCommand (/home/…/logsene-js/node_modules/yargs/lib/command.js:242:26)
    at Object.parseArgs [as _parseArgs] (/home/…/logsene-js/node_modules/yargs/yargs.js:1113:24)
    at Object.parse (/home/…/logsene-js/node_modules/yargs/yargs.js:575:25)
    at exports.main (/home/…/logsene-js/node_modules/mocha/lib/cli/cli.js:68:6)
    at Object.<anonymous> (/home/…/logsene-js/node_modules/mocha/bin/mocha:162:29)
    at Module._compile (node:internal/modules/cjs/loader:1105:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1159:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Module._load (node:internal/modules/cjs/loader:827:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:77:12)
    at node:internal/main/run_main_module:17:47

Node.js v18.1.0

Updating the ip dependency (to v1.1.8) fixes the module loading issue, but there are still test failures (that don't occur with Node 16 on the same system):

  18 passing (2m)
  2 failing

  1) Accept dynamic index name function
       generates index name per document:
     Error: Timeout of 25000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/…/logsene-js/test/test.js)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

  2) Using _index from message + remove _index field from message
       generates index name per document:
     Error: Timeout of 25000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/home/…/logsene-js/test/test.js)
      at listOnTimeout (node:internal/timers:564:17)
      at process.processTimers (node:internal/timers:507:7)

Creation typings to provide TypeScript support

In order to import and use this within TypeScript projects @types are needed.

Add error parameter in callback when .send() fails.

logsene-js/index.js

Line 481 in c42f5ff

}

If the .send() method fails there should be an error returned in the callback for easy error handling.

Elasticsearch 8: Action/metadata line [1] contains an unknown parameter [_type]

I'm trying to use logagent with a newly-installed Elastic 8.5.3 and getting the following error:

logagent_1  | 2023-01-23T11:24:38.977Z v3.1.3 pid[7] Error in Elasticsearch request:  source="logsene-js" err={"message":"HTTP status code:400","httpStatus":400,"httpBody":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}","url":"http://192.168.0.10:9001/\"docker\"/_bulk"} /  message="HTTP status code:400" httpStatus=400 httpBody="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" url="http://192.168.0.10:9001/\"docker\"/_bulk"

The error indicates source logsene-js, hence posting the issue here rather than in logagent-js.

The "_type" parameter seems to be used at:
https://github.com/sematext/logsene-js/blob/master/index.js#L419

  if (elasticsearchDocId !== null) {
    this.bulkReq.write(stringifySafe({ 'index': { '_index': _index, '_id': String(elasticsearchDocId), '_type': type || this.type } }) + '\n')
  } else {
    this.bulkReq.write(stringifySafe({ 'index': { '_index': _index, '_type': type || this.type } }) + '\n')
  }

The "_type" parameter was deprecated for bulk operations as of Elasticsearch 8.0: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html

If my diagnosis is correct, could this be updated to be compatible with Elasticsearch 8.0+?

Environment
Running latest docker image:
docker pull sematext/logagent

My docker-compose.yml:

logagent:
  image: 'sematext/logagent:latest'
  environment:
    - LOGS_TOKEN="docker"
    - LOGS_RECEIVER_URL="http://192.168.0.10:9001"
  cap_add:
    - SYS_ADMIN
  restart: always
  volumes:
    - '/var/run/docker.sock:/var/run/docker.sock'