GithubHelp home page GithubHelp logo

signalfx-nodejs's Introduction

ℹ️  SignalFx was acquired by Splunk in October 2019. See Splunk SignalFx for more information.

GitHub release (latest by date) GitHub branch checks state npm node-current

Node.js client library for SignalFx

This is a programmatic interface in JavaScript for SignalFx's metadata and ingest APIs. It is meant to provide a base for communicating with SignalFx APIs that can be easily leveraged by scripts and applications to interact with SignalFx or report metric and event data to SignalFx.

Installation

To install using npm:

$ npm install signalfx

Supported Node.js versions

Version Node.js
8.x.x >=12.10.0 <=20
7.4.x >=8.0.0 <18
7.3.1 >=8.0.0 <11

Usage

API access token

To use this library, you need a SignalFx access token. When using the ingest client you will need to specify your organization's access token. For the SignalFlow Client, either an organization access token or a user API token may be used. For more information on access tokens, see the API's Authentication documentation.

Create client

There are two ways to create an ingest client object:

  • The default constructor Ingest. This constructor uses Protobuf to send data to SignalFx. If it cannot send Protobuf, it falls back to sending JSON.
  • The JSON constructor IngestJson. This constructor uses JSON format to send data to SignalFx.
var signalfx = require("signalfx");

// Create default client
var client = new signalfx.Ingest("ORG_TOKEN", { options });
// or create JSON client
var clientJson = new signalfx.IngestJson("ORG_TOKEN", { options });

Object options is an optional map and may contains following fields:

  • enableAmazonUniqueId - boolean, false by default. If true, library will retrieve Amazon unique identifier and set it as AWSUniqueId dimension for each datapoint and event. Use this option only if your application deployed to Amazon
  • dimensions - object, pre-defined dimensions for each datapoint and event. This object has key-value format { dimension_name: dimension_value, ...}
  • ingestEndpoint - string, custom url to send datapoints in format http://custom.domain/api/path
  • timeout - number, sending datapoints timeout in ms (default is 5000ms)
  • batchSize - number, batch size to group sending datapoints
  • userAgents - array of strings, items from this array will be added to 'user-agent' header separated by comma
  • proxy - object, defines an address and credentials for sending metrics through a proxy server, it has the following format:
    {
      protocol: 'http(s)',
      host: '127.0.0.1',
      port: 1234,
      auth: {
        username: '<username>',
        password: '<password>'
      }
    },

Configuring the ingest endpoint

If the ingestEndpoint is not set manually, this library uses the us0 realm by default. If you are not in this realm, you will need to explicitly set the endpoint urls above. To determine if you are in a different realm and need to explicitly set the endpoints, check your profile page in the SignalFx web application.

Reporting data

This example shows how to report metrics to SignalFx, as gauges, counters, or cumulative counters.

var signalfx = require('signalfx');

var client = new signalfx.Ingest('ORG_TOKEN');

client.send({
           cumulative_counters:[
             {  metric: 'myfunc.calls_cumulative',
                value: 10,
                timestamp: 1442960607000},
             ...
           ],
           gauges:[
             {  metric: 'myfunc.time',
                value: 532,
                timestamp: 1442960607000},
             ...
           ],
           counters:[
             {  metric: 'myfunc.calls',
                value: 42,
                timestamp: 1442960607000},
             ...
           ]});

The timestamp must be a millisecond precision timestamp; the number of milliseconds elapsed since Epoch. The timestamp field is optional, but strongly recommended. If not specified, it will be set by SignalFx's ingest servers automatically; in this situation, the timestamp of your datapoints will not accurately represent the time of their measurement (network latency, batching, etc. will all impact when those datapoints actually make it to SignalFx).

Sending multi-dimensional data

Reporting dimensions for the data is also optional, and can be accomplished by specifying a dimensions parameter on each datapoint containing a dictionary of string to string key/value pairs representing the dimensions:

var signalfx = require('signalfx');

var client = new signalfx.Ingest('ORG_TOKEN');

client.send({
          cumulative_counters:[
            { 'metric': 'myfunc.calls_cumulative',
              'value': 10,
              'dimensions': {'host': 'server1', 'host_ip': '1.2.3.4'}},
            ...
          ],
          gauges:[
            { 'metric': 'myfunc.time',
              'value': 532,
              'dimensions': {'host': 'server1', 'host_ip': '1.2.3.4'}},
            ...
          ],
          counters:[
            { 'metric': 'myfunc.calls',
              'value': 42,
              'dimensions': {'host': 'server1', 'host_ip': '1.2.3.4'}},
            ...
          ]});

Sending events

Events can be send to SignalFx via the sendEvent function. The event param objects must be specified. Event param object is an optional map and may contains following fields:

  • eventType (string) - Required field. The event type (name of the event time series).
  • category (int) - the category of event. Choose one from EVENT_CATEGORIES list. Different categories of events are supported.Available categories of events are USER_DEFINED, ALERT, AUDIT, JOB, COLLECTD, SERVICE_DISCOVERY, EXCEPTION. For mode details see proto/signal_fx_protocol_buffers.proto file. Value by default is USER_DEFINED
  • dimensions (dict) - a map of event dimensions, empty dictionary by default
  • properties (dict) - a map of extra properties on that event, empty dictionary by default
  • timestamp (int64) - a timestamp, by default is current time Also please specify event category: for that get option from dictionary client.EVENT_CATEGORIES.
var signalfx = require("signalfx");

var client = new signalfx.Ingest("ORG_TOKEN");

client.sendEvent({
  category: "[event_category]",
  eventType: "[event_type]",
  dimensions: {
    host: "myhost",
    service: "myservice",
    instance: "myinstance",
  },
  properties: {
    version: "event_version",
  },
  timestamp: timestamp,
});

Examples

Complete code example for Reporting data

var signalfx = require("signalfx");

var myToken = "ORG_TOKEN";

var client = new signalfx.Ingest(myToken);
var gauges = [
  {
    metric: "test.cpu",
    value: 10,
  },
];

var counters = [
  {
    metric: "cpu_cnt",
    value: 2,
  },
];

client.send({ gauges: gauges, counters: counters });

Complete code example for Sending events

var signalfx = require("signalfx");

var myToken = "[ORG_TOKEN]";

var client = new signalfx.Ingest(myToken);

var eventCategory = client.EVENT_CATEGORIES.USER_DEFINED;
var eventType = "deployment";
var dimensions = {
  host: "myhost",
  service: "myservice",
  instance: "myinstance",
};
var properties = { version: "[EVENT-VERSION]" };

client.sendEvent({
  category: eventCategory,
  eventType: eventType,
  dimensions: dimensions,
  properties: properties,
});

See example/general_usage.js for a complete code example for Reporting data. Set your SignalFx token and run example

$ SPLUNK_ACCESS_TOKEN=xxx SPLUNK_REALM=xxx node example/general_usage.js

Log level

The default log level is info. You can override it by setting SFX_CLIENT_LOG_LEVEL environment variable. Valid values are winston log levels: error, warn, info, http, verbose, debug and silly.

SignalFlow API

SignalFlow is SignalFx's real-time analytics computation language. The SignalFlow API allows SignalFx users to execute real-time streaming analytics computations on the SignalFx platform. For more information, head over to our Developers documentation:

API access token

The SignalFlow cilent accepts either an Organization Access Token or a User API Token. For more information on access tokens, see the API's Authentication Documentation.

SignalFlow

Configuring the Signalflow websocket endpoint

If the websocket endpoint is not set manually, this library uses the us0 realm by default. If you are not in this realm, you will need to explicitly set the endpoint urls above. To determine if you are in a different realm and need to explicitly set the endpoints, check your profile page in the SignalFx web application.

Examples

Complete code example for executing a computation

var signalfx = require('signalfx');

var wsCallback = function(evt) {
    console.log('Hello, I'm a custom callback: ' + evt);
}

var myToken = '[ACCESS_TOKEN]';
var options = {'signalflowEndpoint': 'wss://stream.{REALM}.signalfx.com',
               'apiEndpoint': 'https://api.{REALM}.signalfx.com',
               'webSocketErrorCallback': wsCallback
              };

var client = new signalfx.SignalFlow(myToken, options);

var handle = client.execute({
            program: "data('cpu.utilization').mean().publish()",
            start: Date.now() - 60000,
            stop: Date.now() + 60000,
            resolution: 10000,
            immediate: false});

handle.stream(function(err, data) { console.log(data); });

Object options is an optional map and may contains following fields:

  • signalflowEndpoint - string, wss://stream.us0.signalfx.com by default. Override this if you are in a different realm than us0.
  • apiEndpoint - string, https://api.us0.signalfx.com by default. Override this if you are in a different realm than us0.
  • webSocketErrorCallback - function, Throws an Error event by default. Override this if you want to handle a websocket error differently.

Note: A token created via the REST API is necessary to use this API. API Access tokens intended for ingest are not allowed.

API Options

Parameters to the execute method are as follows :

  • program (string) - Required field. The signalflow to be run.
  • start (int | string) - A milliseconds since epoch number or a string representing a relative time : e.g. -1h. Defaults to now.
  • stop (int | string) - A milliseconds since epoch number or a string representing a relative time : e.g. -30m. Defaults to infinity.
  • resolution (int) - The interval across which to calculate, in 1000 millisecond intervals. Defaults to 1000.
  • maxDelay (int) - The maximum time to wait for a datapoint to arrive, in 10000 millisecond intervals. Defaults to dynamic.
  • bigNumber (boolean) - True if returned values require precision beyond MAX_SAFE_INTEGER. Returns all values in data messages as bignumber objects as per https://www.npmjs.com/package/bignumber.js Defaults to false.
  • immediate (boolean) - Whether to adjust the stop timestamp so that the computation doesn't wait for future data to be available.

Computation Objects

The returned object from an execute call possesses the following methods:

  • stream (function (err, data)) - accepts a function and will call the function with computation messages when available. It returns multiple types of messages, detailed below. This follows error first callback conventions, so data is returned in the second argument if no errors occurred.
  • close () - terminates the computation.
  • get_known_tsids () - gets all known timeseries ID's for the current computation
  • get_metadata (string) - gets the metadata message associated with the specific timeseries ID.

Stream Message Types

  • Metadata
{
  type : "metadata",
  channel : "<CHID>",
  properties : {
    sf_key : [<String>]
    sf_metric: <String>
    ...
  },
  tsId : "<ID>"
}
  • Data
{
  type : "data",
  channel : "<CHID>",
  data : [
    {
      tsId : "<ID>",
      value : <Number>
    },
    ...
  ],
  logicalTimestampMs : <Number>
}
  • Event
{
  tsId : "<ID>",
  timestampMs: 1461353198000,
  channel: "<CHID>",
  type: "event",
  properties: {
    incidentId: "<ID>",
    "inputValues" : "{\"a\":4}",
    was: "ok",
    is: "anomalous"
  }
}

More information about messages can be found at https://developers.signalfx.com/v2/docs/stream-messages-specification

Usage in a Web Browser

The signalflow client can be built for usage in a browser. This is accomplished via the following commands.

$ npm install
$ npm run build:browser
The output can be found at ./build/signalfx.js

It can then be loaded as usual via a script tag

<script src="build/signalfx.js" type="text/javascript"></script>

Once loaded, a signalfx global will be created(window.signalfx). Note that only the SignalFlow package is included in this built file.

Browser Usage Example using D3

First ensure your current working directory is the root of the nodejs repository clone, then do the following :

Make the following changes to example/index.html

replace 'ACCESS_TOKEN' with your own token.
replace 'cpu.utilization' with an appropriate metric as necessary.

Execute the following commands

$ npm install
$ gulp browserify
$ node example/server.js

Finally, open http://localhost:8888/example/index.html

License

Apache Software License v2 © SignalFx

signalfx-nodejs's People

Contributors

9park avatar almightyoatmeal avatar amoshgabadi avatar beccatortell avatar bkarthiksf avatar cep21 avatar charless-splunk avatar dakimsplunk avatar dependabot[bot] avatar djmitche avatar edhusplunk avatar jrcamp avatar jtmal-signalfx avatar jtmalinowski avatar junyu-w avatar kchengsf avatar ljht avatar mdubbyap avatar mmusik avatar mpetazzoni avatar n3m-sfx avatar owais avatar seemk avatar sfishel-splunk avatar sheremetat avatar ssingamneni-sfx avatar tedoc2000 avatar theletterf avatar tysonandre avatar unsetbit avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

signalfx-nodejs's Issues

Cannot install due to node engine restriction in package.json

I have been using this module for a while on v7.2.0, but lately my yarn install calls fail with this error

error [email protected]: The engine "node" is incompatible with this module. Expected version ">=8.0.0 <11". Got "14.17.0"

I am guessing that a more recent release has changed the engines.node field to have a max value of 11, but node v8 and v10 are both already EOL and v16 is about to enter LTS.

I would advise updating the required node version to be >= 8.0.0 with no max version, or set it to >=8.0.0 <17.

Getting timeouts when calling from Node Lambda

We have an AWS Lambda function that uses the SignalFx node client to post metrics to our account. Lately I've been seeing an increased number of these errors in our Lambda's CloudWatch logs:

error: Failed to send datapoint: undefined Error: ETIMEDOUT
 at null._onTimeout (/var/task/node_modules/signalfx/node_modules/request/request.js:846:19)
 at Timer.listOnTimeout (timers.js:92:15)

The incidence of these errors relative to overall load is not high, but we have seen a spike in these errors recently.

SignalFxClient queue will grow in size if the average send() datapoints length exceeds batchSize, leaking memory

For example, if an application calls SignalFxClient.send() periodically with 400 datapoints and the default batchSize of 300, it will leak memory because at most one HTTP post is made per call to send():

  1. 400 items will get added to signalFxClient.queue
  2. send() will lead to a single call to startAsyncSend(), and 300 (batchSize) items will get removed from signalFxClient.queue, leaving 100 items in signalFxClient.queue
  3. When this is repeated, queue will grow by 100 elements each time, leaking memory

// Send post request in separate thread
var datapointsList = [];
while (_this.queue.length !== 0 && datapointsList.length < _this.batchSize) {
datapointsList.push(_this.queue.shift());
}
if (datapointsList.length > 0) {
var dataToSend = _this._batchData(datapointsList);
if (dataToSend && dataToSend.length > 0) {
var url = _this.ingestEndpoint + '/' + _this.INGEST_ENDPOINT_SUFFIX;
return _this.post(dataToSend, url, _this.getHeaderContentType());
}
}

exports.DEFAULT_BATCH_SIZE = 300;// Will wait for this many requests before posting

Possible Solutions

  1. After successfully sending data points (promise resolves), call startAsyncSend() again if this.queue.length >= this.batchSize
    (response) => response.data,
  2. Introduce a max queue size and log a warning if this is exceeded
  3. Chunk calls into batchSize and call SignalFx concurrently
  4. Allow calls to be made in sizes that are larger than batchSize (This may cause problems if SignalFX endpoints reject http requests where the body size in bytes is too large; I haven't tried this)

This affects 7.x and 8.0.0beta releases (I didn't check earlier release lines)

Pin protobufjs dependency to a secure version

A project I work on lists this package as a dependency. As of recently, npm install has started to throw security warnings relating to a dependency pulled in through the signalfx package.

Please update your dependencies list to require a secure version of protobufjs to clear up this warning.

>$ npm install
<REDACTED unrelated deprecations etc.>

added ### packages from ### contributors and audited 3435 packages in 12.892s
found 1 moderate severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details
>$ npm audit

                       === npm audit security report ===

┌──────────────────────────────────────────────────────────────────────────────┐
│                                Manual Review                                 │
│            Some vulnerabilities require your attention to resolve            │
│                                                                              │
│         Visit https://go.npm.me/audit-guide for additional guidance          │
└──────────────────────────────────────────────────────────────────────────────┘
┌───────────────┬──────────────────────────────────────────────────────────────┐
│ Moderate      │ Denial of Service                                            │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Package       │ protobufjs                                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Patched in    │ >=5.0.3 < 6.0.0 || >=6.8.6                                   │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Dependency of │ signalfx                                                     │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ Path          │ signalfx > protobufjs                                        │
├───────────────┼──────────────────────────────────────────────────────────────┤
│ More info     │ https://nodesecurity.io/advisories/605                       │
└───────────────┴──────────────────────────────────────────────────────────────┘
found 1 moderate severity vulnerability in 3435 scanned packages
  1 vulnerability requires manual review. See the full report for details.

Node Engine Restrictions

We are attempting to install 8.0.0-beta.6 with the current Node LTS v20.11.1 with npm v10.2.4 but getting an unsupported engine error from npm:

Unsupported engine {
  package: '[email protected]',
  required: { node: '>=12.10 <=20', npm: '>=7.0 <10' },
  current: { node: 'v20.11.1', npm: '10.2.4' }
}

I'm guessing because the max values of node and npm listed are 20 and 10 respectively. Is it possible to get these updated to support the current LTS version of Node/npm?

No support for REST API

The Python client nicely supports the SignalFX REST API, but the node client appears not to.

Feature request: Make enableAmazonUniqueId work for fargate(ECS)

EC2 is already implemented and uses https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-identity-documents.html , which does not exist for fargate

The IPv4 address 169.254.169.254 is a link-local address and is valid only from the instance.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

https://stackoverflow.com/a/75993784 https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-metadata-endpoint-v4-fargate.html#task-metadata-endpoint-v4-fargate-examples has fields such as dockerId/privateId. dockerId seems the most similar to instanceId.

The environment variable ECS_CONTAINER_METADATA_URI_V4 is set in AWS fargate platform 1.4.0+

DockerId
The Docker ID for the container.
When you use Fargate, the id is a 32-digit hex followed by a 10 digit number.

When querying the ${ECS_CONTAINER_METADATA_URI_V4} endpoint you are returned only metadata about the container itself. The following is an example output.

{
    "DockerId": "cd189a933e5849daa93386466019ab50-2495160603",

Aside: Multiple ECS containers can run on individual EC2 instances. I haven't checked what the endpoint currently does in that case. To preserve existing behavior, 169.254.169.254 (exports.AWS_UNIQUE_ID_URL) could be tried first: https://cloudonaut.io/ecs-vs-fargate-whats-the-difference/

Winston version 1.0.1 uses an outdated dependency pkginfo that appears to be incompatible with Webpack

The Node.js client library for SignalFx uses version 1.0.1 of Winston. Version 2.3.0 of the Winston library includes a fix for an issue related to an outdated pkginfo dependency (the pull request for Winston that resolved this issue is here: winstonjs/winston#931). Versions of Winston prior to 2.3.0 are incompatible with any Node application bundled via Webpack, since pkginfo cannot find package.json from the bundled output.

Currently, using the Node.js client library for SignalFx in a Node application bundled via Webpack causes failure at the start script, node dist/index:

Could not find package.json up from: /
at Function../node_modules/pkginfo/lib/pkginfo.js.pkginfo.find
at Function../node_modules/pkginfo/lib/pkginfo.js.pkginfo.read
at ./node_modules/pkginfo/lib/pkginfo.js.module.exports
at Object.
at Object../node_modules/pkginfo/lib/pkginfo.js
at webpack_require
at fn
at Object.
at Object../node_modules/winston/lib/winston.js
at webpack_require

Circular dependency in Node 14.x

Node 14.5.0 complains about a circular dependency due to winston module.

(node:18613) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
    at emitCircularRequireWarning (internal/modules/cjs/loader.js:824:11)
    at Object.get (internal/modules/cjs/loader.js:838:5)
    at Object.exports.setLevels (/home/e-src/2020-07-11-20-20-21/node_modules/signalfx/node_modules/winston/lib/winston/common.js:35:14)
    at Object.<anonymous> (/home/e-src/2020-07-11-20-20-21/node_modules/signalfx/node_modules/winston/lib/winston.js:84:8)
    at Module._compile (internal/modules/cjs/loader.js:1201:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1221:10)
    at Module.load (internal/modules/cjs/loader.js:1050:32)
    at Function.Module._load (internal/modules/cjs/loader.js:938:14)
    at Module.require (internal/modules/cjs/loader.js:1090:19)
    at require (internal/modules/cjs/helpers.js:75:18)

Logger override

Is it possible to override SignalFx's logger? Our app has a logger that forwards to a centralized logging platform (you know, Splunk :)). We'd like to intercept SignalFX failures and add some additional context before actually logging them. As is, we just see errors like this (for example):

error: Failed to send datapoint: 401 Unauthorized 

And without digging in, it's not obvious that it's even the SignalFx library logging these errors.

Search API Support

We are working on a tool where we need to get the search results data as part of API response. How can we achieve it?

New Search:

index=logs* source IN (service1,service2,service3) name=helloworld machine=m14 | stats count p95(duration) as P95 avg(calDuration) as Avg p99(duration) as P99 by type name

Unhandled WS error crashes program when stream endpoint becomes unreachable

When using the Signalflow part of this library, when the endpoint becomes unreachable (network outage or similar) it will crash the application due to an unhandled error:

Error: getaddrinfo ENOTFOUND stream.eu0.signalfx.com stream.eu0.signalfx.com:443
    at errnoException (dns.js:50:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:92:26)
npm ERR! code ELIFECYCLE
npm ERR! errno 1

The source of the error is https://github.com/signalfx/signalfx-nodejs/blob/master/lib/client/signalflow/request_manager.js#L154-L160 there is no error handler being set on the WS connection.

To reproduce:

  1. Start a signalflow computation via websockets
  2. Disconnect from internet
  3. wait approx 20 seconds

Expected:
The error would be propagated up to the top level in some way so the application using this library can decide what to do with it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.