clearbit / clearbit-node Goto Github PK

I'm trying to respect the Rate Limiting defined in the docs: https://clearbit.com/docs#rate-limiting

Catching the error thrown on a rate_limit call I get:

{"message":"Rate limit exceeded. Limit is 5 concurrent requests.","type":"rate_limit","body":{"error":{"type":"rate_limit","message":"Rate limit exceeded. Limit is 5 concurrent requests."}},"statusCode":400}

There doesn't seem to be any easy access to the response headers here, any chance I could ask you to add this please. I'll just use an exponential backoff (maxing out at 8s) for now but would be great to get access to the Retry-After header in particular to optimise the wait period.

Also looks like either the API is throwing the wrong code or the client lib is transforming it to 400, the docs suggest it should be returning a 429.

Thanks,
Alex

Just parsing through the various errors we get back from a bunch of calls and looks like "PersonNotFound" returns with:

{"message":"Resource not found"}

All the other errors seem like they have an error.type property I can check and/or a statusCode attribute. Any chance we can add this in to make this one consistent, I'll match against the message for now :)

Thanks,
Alex

this code (node 7.10.1):

const Client   = require('clearbit').Client;
const clearbit = new Client({key: process.env.clearbitKey});

let Organization = clearbit.Company;
Organization.find({domain: req.body.url).then((company) => {
    console.log('Name: ', company.name);
  }).catch(Company.QueuedError, (err) => {
    console.log(err); // Company is queued
  }).catch(Company.NotFoundError, (err) => {
    console.log(err); // Company could not be found
  }).catch((err) => {
    console.log('Bad/invalid request, unauthorized, Clearbit error, or failed request');
    console.log('err');
  });

throws this error:

➜  git:(feature/clearbit) ✗ node clearbit.js
app.js:10
Organization.find({domain: req.body.url).then((company) => {
                                       ^
SyntaxError: Unexpected token )
    at createScript (vm.js:53:10)
    at Object.runInThisContext (vm.js:95:10)
    at Module._compile (module.js:543:28)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3)
    at Module.runMain (module.js:605:10)
    at run (bootstrap_node.js:427:7)
    at startup (bootstrap_node.js:151:9)

either the docs are outdated or the package is broken

Hi,

I am trying to do npm install clearbit in AWS EC2 t3 linux instance and I am getting the below error.

Command failed: git clone --depth=1 -q -b master git://github.com/clearbit/needle.git /root/.npm/_cacache/tmp/git-clone-d0e831ca npm ERR! fatal: could not create leading directories of '/root/.npm/_cacache/tmp/git-clone-d0e831ca': Permission denied

I tried to do sudo npm instal too which is not recommended.

node version - 10.6.0
npm version - 6.1.0

npm ERR! code ENOGIT
npm ERR! Error while executing:
npm ERR! undefined ls-remote -h -t ssh://[email protected]/clearbit/needle.git
npm ERR! 
npm ERR! undefined
npm ERR! No git binary found in $PATH
npm ERR!
npm ERR! Failed using git.
npm ERR! Please check if you have git installed and in your PATH.

We are using docker and does not have git on it.

Hi guys,

I did a very basic integration using this API but the size of the bundle is too big, I think because of the dependencies.
See the following screenshot and below it, my package.json file:

{
  "name": "webpack-starter",
  "version": "1.0.0",
  "description": "A light foundation for your next frontend project based on webpack.",
  "scripts": {
    "build": "webpack --config webpack/webpack.config.prod.js  --colors",
    "start": "webpack-dev-server --open --config webpack/webpack.config.dev.js"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/wbkd/webpack-starter.git"
  },
  "keywords": [
    "webpack",
    "startkit",
    "frontend",
    "es6",
    "javascript",
    "webdev"
  ],
  "author": "webkid.io",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/wbkd/webpack-starter/issues"
  },
  "devDependencies": {
    "@babel/core": "^7.4.0",
    "@babel/plugin-proposal-class-properties": "^7.4.0",
    "@babel/plugin-syntax-dynamic-import": "^7.2.0",
    "@babel/preset-env": "^7.4.2",
    "babel-loader": "^8.0.5",
    "clean-webpack-plugin": "^2.0.2",
    "copy-webpack-plugin": "^5.0.3",
    "css-loader": "^3.0.0",
    "eslint": "^5.16.0",
    "eslint-loader": "^2.1.2",
    "file-loader": "^4.0.0",
    "html-webpack-plugin": "^4.0.0-beta.5",
    "mini-css-extract-plugin": "^0.7.0",
    "node-sass": "^4.12.0",
    "sass-loader": "^7.1.0",
    "style-loader": "^0.23.1",
    "webpack": "^4.29.6",
    "webpack-bundle-analyzer": "^3.3.2",
    "webpack-cli": "^3.3.0",
    "webpack-dev-server": "^3.2.1",
    "webpack-merge": "^4.2.1"
  },
  "dependencies": {
    "@babel/polyfill": "^7.4.0",
    "clearbit": "^1.3.4",
    "core-js": "^3.0.1"
  }
}

is there anything I can do to decrease the size?

my script is just a js with this example from the docs:

var clearbit = require('clearbit')('sk_<redacted>');

clearbit.Enrichment.find({email: '[email protected]', stream: true})
  .then(function (response) {
    var person  = response.person;
    var company = response.company;

    console.log('Name: ', person && person.name.fullName);
  })
  .catch(function (err) {
    console.error(err);
  });

I guess this is because it's meant to be used in node and not in the front end. But it can probably be optimised a little. Like it's only using three methods of lodash and still imports *

Hi, I've just had this error occur in production (node.js 9.3.0):

TypeError: this.NotFoundError is not a constructor
1
at Function.<anonymous> (/app/node_modules/clearbit/src/resource.js line 25 col 13)
throw new this.NotFoundError(this.name + ' not found');
2
at Function.__NR_wrappedThenHandler (/app/node_modules/newrelic/lib/instrumentation/promise.js line 344 col 23)
return fn.apply(this, arguments)
3
at Function.tryCatcher (/app/node_modules/clearbit/node_modules/bluebird/js/main/util.js line 26 col 23)
return target.apply(this, arguments);
4
at CatchFilter.doFilter (/app/node_modules/clearbit/node_modules/bluebird/js/main/catch_filter.js line 52 col 40)
var ret = tryCatch(cb).call(boundTo, e);
5
at CatchFilter.tryCatcher (/app/node_modules/clearbit/node_modules/bluebird/js/main/util.js line 26 col 23)
return target.apply(this, arguments);
6
at Promise._settlePromiseFromHandler (/app/node_modules/clearbit/node_modules/bluebird/js/main/promise.js line 510 col 31)
x = tryCatch(handler).call(receiver, value);
7
at Promise._settlePromiseAt (/app/node_modules/clearbit/node_modules/bluebird/js/main/promise.js line 584 col 18)
this._settlePromiseFromHandler(handler, receiver, value, promise);
8
at Promise._settlePromises (/app/node_modules/clearbit/node_modules/bluebird/js/main/promise.js line 700 col 14)
this._settlePromiseAt(i);
9
at Async._drainQueue (/app/node_modules/clearbit/node_modules/bluebird/js/main/async.js line 123 col 16)
fn._settlePromises();
10
at Async._drainQueues (/app/node_modules/clearbit/node_modules/bluebird/js/main/async.js line 133 col 10)
this._drainQueue(this._normalQueue);
11
at Immediate.Async.drainQueues [as _onImmediate] (/app/node_modules/clearbit/node_modules/bluebird/js/main/async.js line 15 col 14)
self._drainQueues();
12
at runCallback (timers.js line 773 col 18)
13
at tryOnImmediate (timers.js line 734 col 5)
14
at processImmediate [as _immediateCallback] (timers.js line 711 col 5)

I've tried and failed to reproduce it in a test setting, but then I stumbled upon petkaantonov/bluebird#973 (comment) -- and it seems like it could be due to some non-Bluebird promise slipping in somewhere, rendering .bind(this) ineffective:

Would you accept a patch that avoids the .bind(this) construct? Would be easy to avoid it with arrow functions (node.js 4+) or the good old var that = this; trick if older environments must be supported.

Looks like the clearbit client is using a forked copy of needle for browserify support: clearbit/needle#1. Upstream needle added this support in tomas/needle#146, so can the client use the upstream version of needle to avoid these npm errors?

npm ERR! missing: needle@github:clearbit/needle#84d28b5f2c3916db1e7eb84aeaa9d976cc40054b, required by [email protected]

Hey,

Awesome library!

For whatever reason, access has stopped working for me. I'm passing in a valid API key and I've tried both methods of instantiating the library.

The API key works because I tried CURL.

Any help would be appreciated!

Thanks,
Peter

See #50 (comment) , which (emphasis mine) states clearly this library is unmaintained:

Hi Ronan,

Thanks for reaching out again. To best serve our wide customer base, we continue to make regular updates to our product, improving breadth, depth, and accuracy of our data. We follow industry standard prioritization of engineering resources based on impact and urgency. We are not able to dedicate resources to the node.js library.

Best,

<Name censored, I don't want to name names> | Head of Customer Support | Clearbit

Asking customers to make REST calls on their own is totally acceptable, and plenty of web services do so.

But then please don't pretend to paying customers you're maintaining a library, and please state clearly in this lib's README that it's deprecated / unmaintained.

Searching by email using Enrichment.find(email,webookhook,id)

If their is no person object found, but company object found, the response will not contain the original email used in the enrichment api, hence hard to consolidate with our own data.

Thanks.

Hi,

I'm trying to do a filtered catch as show in the documentation. It doesn't seem to work with NameToDomain.NotFoundError.

var NameToDomain = clearbit.NameToDomain;
NameToDomain.find({name: 'UberrebU'})
  .then(function (result) {
    console.log('Domain: ', result.domain);
  })
  .catch(NameToDomain.NotFoundError, function (err) {
    // NEVER GET HERE!
    console.log(err); // Domain could not be found
  })
  .catch(function (err) {
    console.log(err);
    console.log('Bad/invalid request, unauthorized, Clearbit error, or failed request');
  });

When the domain can not be found, it ends up in the last catch block with the following error:

{ ClearbitError: Not Found
    at ClearbitClient.<anonymous> (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/src/client.js:82:22)
    at ClearbitClient.tryCatcher (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/promise.js:508:31)
    at Promise._settlePromiseAt (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/promise.js:584:18)
    at Promise._settlePromises (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/promise.js:700:14)
    at Async._drainQueue (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/Users/brandon/code/new-bonfire-organization/node_modules/clearbit/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:574:20)
    at tryOnImmediate (timers.js:554:5)
    at processImmediate [as _immediateCallback] (timers.js:533:5)
  message: 'Not Found',
  type: 'unknown',
  body: '{\n  "type": "unknown_record",\n  "message": "Unknown company."\n}',
  statusCode: 404 }

https://logo.clearbit.com/apple.com

It gets blocked I think because it has the word "logo" in the url.

Clearbit base and logo domains are in the "Easy List", enabled by default by Adblock Plus.
https://easylist-downloads.adblockplus.org/easyprivacy.txt

I observed the issue using the Autocomplete API (https://autocomplete.clearbit.com/v1/companies/suggest?query=something).

Linked to #35 .

Are there typescript typings?

is the webhook_id passed back in the webhook as a query param, body or in the headers? could not locate it in all 3 options.

Wrap the Autocomplete API.

Installing clearbit via npm install in heroku causes ssh error. I faced this issue previously it was resolved the next day without any changes. I have attached the logs which it tries to execute.

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://[email protected]/clearbit/needle.git
npm ERR! 

Hi,

I'm doing a daily batch of a couple thousand emails to the Enrichment API. I don't set stream: true and I handle the results through the webhook API.

Here's a snippet of how I'm doing this:

  clearbit.Enrichment.find({
    email: user.email
  }).then(function(response) {
    var person = response.person;
    var company = response.company;
    // Let the webhook handle inserting
    done();

    //insertData(db, done, user, person, company);

  }).catch(clearbit.Enrichment.QueuedError, function(err) {
    done();
    // This is fine
    // Our webhook will get the data
  }).catch(function(err) {
    done();
    console.error('ERROR LOOKING UP', err);
  })

What I'm noticing is, after a couple hundred calls to clearbit.Enrichment.find, the calls start erroring out with this stack trace:

ERROR LOOKING UP { [Error: socket hang up]
  cause: { [Error: socket hang up] code: 'ECONNRESET' },
  isOperational: true,
  code: 'ECONNRESET' }
Trace
    at Function.<anonymous> (/app/bin/worker.js:56:13)
    at Function.tryCatcher (/app/node_modules/clearbit/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/app/node_modules/clearbit/node_modules/bluebird/js/main/promise.js:510:31)
    at Promise._settlePromiseAt (/app/node_modules/clearbit/node_modules/bluebird/js/main/promise.js:584:18)
    at Promise._settlePromises (/app/node_modules/clearbit/node_modules/bluebird/js/main/promise.js:700:14)
    at Async._drainQueue (/app/node_modules/clearbit/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/app/node_modules/clearbit/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/app/node_modules/clearbit/node_modules/bluebird/js/main/async.js:15:14)
    at processImmediate [as _immediateCallback] (timers.js:371:17)

Related to #3?

When trying to find a person, I get a connection issue.

Failing test case:

var clearbit = require('clearbit')('KEY_REDACTED');

clearbit.Person.find({email: '[email protected]'})
  .then(function (person) {
    console.log('Name: ', person.name.fullName);
  })
  .catch(clearbit.Person.QueuedError, function (err) {
    // Lookup is queued - try again later
    console.log(err);
  })
  .catch(clearbit.Person.NotFoundError, function (err) {
    // Person could not be found
    console.log(err);
  })
  .catch(function (err) {
    console.error(err);
  });

and the error

{ name: 'OperationalError',
  message: 'socket hang up',
  cause: { [Error: socket hang up] code: 'ECONNRESET' },
  stack: 'Error: socket hang up\n    at createHangUpError (http.js:1472:15)\n    at CleartextStream.socketCloseListener (http.js:1522:23)\n    at CleartextStream.EventEmitter.emit (events.js:117:20)\n    at tls.js:696:10\n    at process._tickCallback (node.js:415:13)' }

but when I just connect using tls.connect() (or request even) - it works.

require('tls').connect(443, 'person.clearbit.com', function() {
  this.write('GET /v1/people/email/[email protected] HTTP/1.1\r\nAuthorization: Basic REDACTED\r\nHost: person.clearbit.com\r\n\r\n');
  this.pipe(process.stdout);
});

Some details on my system:

• Linux vagrant 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
• Linux flavor - Ubuntu 14.04.1 LTS
• nodejs version = 0.10.25
• npm version = 1.3.10
• OpenSSL version = OpenSSL 1.0.1f 6 Jan 2014

Here is a gist of with my Vagrantfile if you'd like to test in the exact same circumstances. https://gist.github.com/jonDowdle/91c930cc3bb6a324676e

There is an API for returning more information, any chance this will support it?

Example: https://person-stream.clearbit.com/v2/combined/find?email=contra%40wearefractal.com (used on this demo page https://clearbit.com/enrichment)

Bluebird and lodash are both currently pinned to version 2, which are now out of date.

Could you look at upgrading these?

Lodash in particular now gives a deprecation notice when running npm install:

[email protected]: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0.

Current version of lodash is 4.15.0. Upgrading from v2 -> 3 is straightforward, upgrading from v3 -> 4 requires checking some aliases which have been removed.

Current version of bluebird is 3.4.1. v2 is deprecated but v3 is a reasonably major upgrade.

When using clearbit to lookup info, a log is added to the console / stdout with the following signature

Making request #1 { protocol: 'https:',
  host: 'company-stream.clearbit.com',
  port: 443,
  path: '/v2/companies/find?domain=google.com',
  method: 'get',
  headers:
   { Accept: '*/*',
     Connection: 'close',
     'User-Agent': 'ClearbitNode/v1.3.3'
...

Currently there is no way to suppress these logs.
I believe the underlying root cause is the way clearbit/needle handles the logs. Since this is a forked version of needle, upgrading the version of needle could solve this issue as the new version of needle uses the debug module.
Code Reference:
node_modules/lib/needle.js line# 25

The best possible option is to provide a way for the clearbit-node client to suppress logs and perform the check in the underlying needle module.

var Client   = require('clearbit').Client
var clearbit = new Client({
   key: 'api_key'
   log: false
})

What's the suggested backoff when I get Person.QueuedError from the API? 2 seconds?

The Needle package you forked is where the issue lies. The way it gets the version out of package.json is incompatible with tools like Webpack and Browserify, which work better using a require() style syntax.

In good news, the owner of the original repo found and fixed this over 2 years ago.
tomas/needle@f7a72f1

In more good news, it's a very straight forward 1 line fix.

First time Clearbit user, just installed this repo. Made my first request and got a PersonQueuedError: lookup queued.

I tried again 10 seconds later, and the promise was resolved successfully.

Is this a common occurrence? How quickly should I retry if a record is showed as 'queued'? Is there any way to enforce real-time look-ups?

My use case is a REST API that needs to send back data in a single request and not subscribe to changes, so I'm hoping there is a way to limit this behaviour.

Thanks in advance!

I am using MongoDB cloud functions (triggers)

After install clearbit as a dependency, all seem to work fine when i use
.then
.catch

but if i use async await, the promise result is [{"isFulfilled":false,"isRejected":false}]"

this seems to be a bluebird issue?

It works on my local machine, but as some users here are also having problems with heroku and other cloud deployments.

Has needle been updated?

(follow-up from #23).

The Clearbit API returns rate limit information inside the HTTP Headers.
AFAIK: this SDK does not surface the header in the response. Is there a way to access those headers (e.g. to figure out the correct backoff) from node-clearbit or do we need to implement our own version to call the API directly and access the headers?

Do you think we should throw an error when the lookup is pending rather than have a 'pending()' function?