appbaseio / appbase-js

appbase.io search client library for JavaScript

appbase-js

appbase-js is a universal JavaScript client library for working with the appbase.io database, for Node.JS and Javascript (browser UMD build is in the dist/ directory); compatible with elasticsearch.js.

An up-to-date documentation for Node.JS API is available at http://docs.appbase.io/javascript/quickstart.

TOC

1. appbase-js: Intro
2. Features
3. Live Examples
4. Installation
5. Docs Manual
6. Other Projects You Might Like

## 1. appbase-js: Intro

appbase-js is a universal JavaScript client library for working with the appbase.io database.

2. Features

It can:

• Index new documents or update / delete existing ones.
• Work universally with Node.JS, Browser, and React Native.

It can't:

• Configure mappings, change analyzers, or capture snapshots. All these are provided by elasticsearch.js - the official Elasticsearch JS client library.

Appbase.io - the database service is opinionated about cluster setup and hence doesn't support the Elasticsearch devops APIs. See rest.appbase.io for a full reference on the supported APIs.

3. Live Examples

Check out the Live interactive Examples at reactiveapps.io.

4. Installation

We will fetch and install the appbase-js lib using npm. 4.0.0-beta is the most current version.

npm install appbase-js

Adding it in the browser should be a one line script addition.

<script
  defer
  src="https://unpkg.com/appbase-js/dist/appbase-js.umd.min.js"
></script>

Alternatively, a UMD build of the library can be used directly from jsDelivr.

To write data to appbase.io, we need to first create a reference object. We do this by passing the appbase.io API URL, app name, and credentials into the Appbase constructor:

var appbaseRef = Appbase({
  url: "https://appbase-demo-ansible-abxiydt-arc.searchbase.io",
  app: "good-books-demo",
  credentials: "c84fb24cbe08:db2a25b5-1267-404f-b8e6-cf0754953c68",
});

OR

var appbaseRef = Appbase({
  url: "https://c84fb24cbe08:[e-mail]",
  app: "good-books-demo",
});

Credentials can also be directly passed as a part of the API URL.

5. Docs Manual

For a complete API reference, check out JS API Ref doc.

6. Other Projects You Might Like

• arc API Gateway for ElasticSearch (Out of the box Security, Rate Limit Features, Record Analytics and Request Logs).

• searchbox A lightweight and performance focused searchbox UI libraries to query and display results from your ElasticSearch app (aka index).

  • Vanilla JS - (~16kB Minified + Gzipped)
  • React - (~30kB Minified + Gzipped)
  • Vue - (~22kB Minified + Gzipped)

• dejavu allows viewing raw data within an appbase.io (or Elasticsearch) app. Soon to be released feature: An ability to import custom data from CSV and JSON files, along with a guided walkthrough on applying data mappings.

• mirage ReactiveSearch components can be extended using custom Elasticsearch queries. For those new to Elasticsearch, Mirage provides an intuitive GUI for composing queries.

• ReactiveMaps is a similar project to Reactive Search that allows building realtime maps easily.

• reactivesearch UI components library for Elasticsearch: Available for React and Vue.

⬆ Back to Top

appbase-js's Issues

Upgrade to ES6

.index() method throws an uncaught error

saying something along the lines of this.response.destroy() not being a function.

search() method returns only first 10 results

i am trying to do a search query on a type chats. I have attached a gist which contains the exported data from dejavu.
https://gist.github.com/f58c46ee458847358780540093546212

It can be seen that there are more than 28 records but even then when i do a search only 10 records are received.

Leaky streams

Usage of global request obj inside stream's stop() has leaky side effects.

Behavior Observation:

I believe this is what is happening (it's my guess based on reading parts of appbase.js; I can be wrong on the specifics, but the problem exists in the lib):

appbase ref object has a field called body, a property of the appbase object which is updated every time the streamingRequest() method is called. This is where it gets interesting -

I have a response object from a streamingRequest() method, I call a stop() on this and in the next execution step call another streamingRequest() [call it 2nd step] method. If my stop() execution is not fast enough, there’s a chance it will use the body field set in the 2nd step.

Reproducing:

Reproducing this is hard, it has to be scripted. The margin of error here is in the order of 2ms. I am testing this with the https://github.com/appbaseio/meetuprsvp-ui code.

Adding two console logs inside the appbase.js ws's stop() code at L12360 (pasted below):

wsRequest.prototype.stop = function stop() {
    console.log("appbase.js", this.request);
    this.client.ws.removeListener('close', this.closeHandler)
    this.client.ws.removeListener('error', this.errorHandler)
    this.client.ws.removeListener('message', this.messageHandler)
    if(this.resultStream.readable) {
        this.resultStream.push(null)
    }
    var unsubRequest = {}
    for(var key in this.request) {
        unsubRequest[key] = this.request[key]
    }
    unsubRequest.unsubscribe = true
    console.log("appbase.js - closing ws request", unsubRequest)
    if(this.unsubscribed !== true) {
        this.client.ws.send(unsubRequest)
    }
    this.unsubscribed = true
}

this.request has different responses in the two console.log() statements; by the time the code execution reaches to the second console.log() - a new streamingRequest() is called which modifies the this.request property.

Possible Fix:

Keeps everything immutable and involves minimal changes to your existing structure

• make request object a part of the returned response stream; and when stop() is called on the stream, use the local request object (instead of the global mutable one) to send the unsubscribe request.

Note: This also has one another nice benefit: as a user, i can debug the request params easily since it's now a part of the responsestream object

URLEncode all Ids in the requests

When an _id field is present in a request - whether it be index(), delete(), update() or even bulk(), can we use URL encoding to ensure that URL unsafe characters can be used in the document _id param?

Related to appbaseio/dejavu#202.

stream response has side-effects

Running two searchStream() methods with the same query object and mutating on the resulting data of one of the streams similarly mutates the resulting data of the other stream.

The lib should use immutable before returning the object to avoid side effects.

Code to reproduce the bug - multiple-streams-bug.js, output shown below:

var Appbase = require('appbase-js')

var appbaseRef = new Appbase({
  url: 'https://RIvfxo1u1:[email protected]',
  appname: 'createnewtestapp01'
})

appbaseRef.searchStream({
  type: ["book", "product", "TestType"],
  body: {
    query: { match_all: {}}
  }
}).on('data', function(res) {
  console.log("response#1: ", res);
  res._source = 'foo';
}).on('error', function(err) {
  console.log("caught a streaming error", err);
})

appbaseRef.searchStream({
  type: ["book", "product", "TestType"],
  body: {
    query: { match_all: {}}
  }
}).on('data', function(res) {
  console.log("response#2: ", res);
}).on('error', function(err) {
  console.log("caught a streaming error: ", err);
})

delete method doesn't invokes on error for unauthorized calls

When using the delete method with read only credentials, it receives the response with a status of 403 but it invokes the on data call instead of the on error call.

Add include: credentials support

Since the underlying request is a fetch we could support this in the coming version

Vue plug-in available?

Is there a Vue plug-in available for this library? I’m current using Vue for my application and think this search capability would be great but I noticed this is mainly for React. What alternatives or workarounds might you recommend?

add a grunt "test' task to run tests

add docs for msearch

Behavior change for supporting search() method on all types

• type field should be optional in the search() method. When no type is specified, it should search on all types in accordance with the multi-type _search API endpoint.
• Additionally, since searchStream() uses * wildcard for applying a stream on all types, search() method should also support type: "*" as a valid way to search on all types.

v1.0.0 on npm package

hi, we want to host your lib on https://cdnjs.com, I saw v1.0.0 on npm package, it looks strange because the latest version is v0.10.8, and there is no v1.0.0 on github.
I want to confirm do we also need to put v1.0.0 on cdnjs?
thank you very much!

cdnjs/cdnjs#8914

Update README and docs

Post v2.3 release (where we apply the custom headers and accept a URL that can have a prefix in its path), we should update the following docs:

• Readme (which is visible on github and npm),
• Official API reference doc over here.

streamDocument() with streamonly method returns an error

Adding streamonly field to the streamDocument() throws a syntax error.

appbaseObj.streamDocument({
  type: "tweet",
  id: "1",
  streamonly: "true"
}).on('data', function(res) {
  console.log("data update: ", res);
}).on('error', function(err) {
  console.log("streaming error: ", err);
})

Stacktrace:

Uncaught SyntaxError: Unexpected token ,conn.onmessage @ appbase.js:483

browser lib throws errors and warnings with create-react-app

Propose we break the browser package out, and investigate the constructor thing (since API should match docs here).

.delete() method should return a 404 error when trying to delete a doc with an invalid id

Code to reproduce the behavior - https://jsfiddle.net/yashshah/cjv3y0t8/.

Since Elasticsearch returns a 404 error in this scenario: https://i.imgur.com/fTRoCKU.png, we should make the lib follow the same logic.

Error: Authentication information is not present. Did you add credentials?

V4.5.1 and V4.5.0, breaks my frontend and backend code.
Last working version 4.4.0
No code has been changed only update appbase-js version or did a npm update on latest version of @appbaseio/reactivesearch

Error: Authentication information is not present. Did you add credentials?

const appbaseDB = Appbase({
  url: <APPBASE_URL>,
  app: <APPBASE_APP>,
  credentials: <APPBASE_KEY>
})

Same on frontend:

<ReactiveBase
        app="APPBASE_APP"
        credentials="APPBASE_KEY">

getStream() throws an error "method does not support streaming." when a document does not exist

A document with a non-existent id throws the following error:

Since get() on a non-existent document returns a JSON object with a field "found" set to false, getStream() should a response message to the effect:

"document with type :type and id :id cannot be found."

Allow plain path for url

I would like to configure an elasticsearch URL of just "/elasticsearch" and allow my server to reverse proxy the HTTP traffic to elasticsearch. Currently the configuration validation code requires the configured URL to include the protocol. This creates a configuration and CORS problem I would like to avoid in the browser by just talking to my main app server and proxying from there. I have so far not found a way to do this due to this validation.

Error: Protocol is not present in url. URL should be of the form https://scalr.api.appbase.io

appbase-js.es.js:128
AppBase appbase-js.es.js:128
index appbase-js.es.js:904
setStore ReactiveBase.js:155
ReactiveBase ReactiveBase.js:67
constructClassInstance23React

.search() method is throwing a ReferenceError

After instantiating Reference object, applying .search() method like below

var response = streamingClient.search({
  "type": "books",
  "body": {
    "query": {"match_all": {}
  }
})

throws

shebang loader issue

json-stream package seems to be using a shebang, figure out a way to either upgrade json-stream or handle the error.

Add custom headers support

Quick start only works with elastic search 5

Hello,

Copy and paste the quick start example only works if the project was created with version 5 of elasticsearch. Is that expected?

Some missing API docs and usage examples

• setHeaders

WebSocket - Uncaught, unspecified 'error' event

Platform: IOS - React Native

Improve promise rejection response

Currently in case of error status >= 400, the promise is rejected with a response object that was already transformed code. It should instead pass the transformed object (data) in this case.

For example, here's the actual error message for a request that went wrong:

{"status":403,"message":"User doesn't have write permission."}

And here is the error that was caught:

{ type: "default", status: 403, ok: false, statusText: "Forbidden", headers: {…}, url: "https://scalr.api.appbase.io/MovieAppFinal/MovieAppFinal/AWNAO8-Q5Q83Zq9GcV4q/_update?", _bodyInit: Blob, _bodyBlob: Blob, bodyUsed: true }

This makes error handling tricky.

Steps to replicate

1. Make a write request with read only credentials
2. Catch the error and log it
3. The caught error and reponse from server should match

Possible incompatibility with react-native

When trying to use appbase-js with react native. It will not install correctly as the dependency hyperquest depends on http and https node modules. There are also modules with this name include with react-native.

Investigate into confirming the issue and finding work arounds to this.

searchStreamToURL with different request objects does not create new object in percolator

Here is my code:

/* Query to check if the new status is posted with specific team_id */
  var requestObject = {
    type: "statuses",
    body: {
      query: {
        match: {
          "team_id": team_id
        }
      }
    }
  };

  var statusBodyObject = {
    "text": "@{{{user_name}}} is working on {{{status}}}"
  }

  var webhookObject = {
    'method': 'POST',
    'url': webhookUrl,
    'body': statusBodyObject
  }

  appbaseRef.searchStreamToURL(requestObject, webhookObject).on('data', function(stream) {
    console.log("Webhook initialized: ", JSON.stringify(stream))
  }).on('error', function(error) {
    console.log("Query error: ", JSON.stringify(error))
  });

Here is my response:

It is always created: false even though I have different team_id value in the request object.

cross-fetch > node-fetch vulnerability

Ref GHSA-r683-j2x4-v87g
Please upgrade cross-fetch package to latest (3.1.5) to fix the issue

bulk index method invokes on data for status 400

Possibly a bug.

Expected behavior - on error gets invoked

ReferenceError on using bulk() method

This is similar to the #2 issue.

appbaseObj.bulk({
  type: "tweet",
  body: [
    // action#1 description
    { index: { _id: 2 } },
    // the JSON data to index
    { "msg": "writing my second tweet!",
      "by": "Ev",
      "using": ["appbase.io", "javascript", "streams"],
      "test": true
    },
    // action#2 description
    { delete: { _id: 2 } },
    // deletion doesn't any further input
  ]
}).on('data', function(res) {
  console.log("successful bulk: ", res);
}).on('error', function(err) {
  console.log("bulk failed: ", err);
})

Stacktrace:

Uncaught ReferenceError: id is not defined(…)bulkService 
@ appbase.js:15bulk 
@ appbase.js:436(anonymous function) 
@ VM989:2InjectedScript._evaluateOn 
@ VM931:904InjectedScript._evaluateAndWrap 
@ VM931:837InjectedScript.evaluate 
@ VM931:693

All lib methods should support URL query parameters

Setting document specific ttl, query specific preference etc are done by URL query parameters. The lib methods like index(), update(), search() should support query parameters.

Custom Type Value Ambiguous Error

Any value other than _doc or _create for the type parameter breaks with a totally ambiguous error message. With code like so provided in the API example
appbaseRef .bulk({ type: 'tweet', ... // rest }).

Why am I getting a 400 error with 'Bad Request'. Can anyone provide any clarity here?

Support optional credentials

credentials when present will set auth headers, otherwise no headers will be used.

Conditionally require babel-polyfill

We should check if babel-polyfill is already present and require it conditionally

Optimize build size

Current build size for a minified version of the library comes to 200KB (~500 KB without minification), GZipped version is ~60KB.

These are the optimization points for reducing bloat from the library:

1. babel-pollyfill contributes ~200KB raw size, or 40% of the size. This can be stripped out in a lite version of the library aimed at native platforms or modern browsers (which have 97% support of ES6 spec).
2. Use tree shaking / rollup to remove other unused parts of the dependency tree. This comes with no sacrifice and should be done.
3. url package (23KB raw) is just used for a parse function. A lightweight alternative can be used for this.
4. Analyze the rest of the lib dependencies for possible optimizations.

Publish the package pre-compiled

Hi,

I've been trying to build a project that consumes reactivesearch+appbase-js but I get this output:

> node scripts/build.js

Creating an optimized production build...
0% compiling
10% building modules
18.04% building modules
26.08% building modules
34.12% building modules
42.16% building modules
50.2% building modules
58.24% building modules
66.28% building modules
71% sealing
72% optimizing
73% basic module optimization
74% module optimization
75% advanced module optimization
76% basic chunk optimization
77% chunk optimization
78% advanced chunk optimization
79% module and chunk tree optimization
80% chunk modules optimization
81% advanced chunk modules optimization
82% module reviving
83% module order optimization
84% module id optimization
85% chunk reviving
86% chunk order optimization
87% chunk id optimization
88% hashing
89% module assets processing
90% chunk assets processing
91% additional chunk assets processing
92% recording
91% additional asset processing
92% chunk asset optimization
94% asset optimization
95% emitting
100%
Failed to compile.

Failed to minify the code from this file:

        ./node_modules/appbase-js/dist/appbase-js.es.js:12

Read more here: http://bit.ly/2tRViJ9

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `node scripts/build.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
[...]

Following the reference link brings me to this description:

npm run build fails to minify

Some third-party packages don't compile their code to ES5 before publishing to npm. This often causes problems in the ecosystem because neither browsers (except for most modern versions) nor some tools currently support all ES6 features. We recommend to publish code on npm as ES5 at least for a few more years.

To resolve this:

• Open an issue on the dependency's issue tracker and ask that the package be published pre-compiled.
  Note: Create React App can consume both CommonJS and ES modules. For Node.js compatibility, it is recommended that the main entry point is CommonJS. However, they can optionally provide an ES module entry point with the module field in package.json. Note that even if a library provides an ES Modules version, it should still precompile other ES6 features to ES5 if it intends to support older browsers.

• Fork the package and publish a corrected version yourself.

• If the dependency is small enough, copy it to your src/ folder and treat it as application code.

In the future, we might start automatically compiling incompatible third-party modules, but it is not currently supported. This approach would also slow down the production builds.

Conclusion

That's why I'm asking here for this package to be published pre-compiled as facebook's create-react-app instructs us.

Thanks

Can't pass credentials via Authorization header

If it is Appbase app and credentials are passed as Authorization header instead of credentials in the config, then it doesn't allow the user to connect.

An error is thrown from the following line
https://github.com/appbaseio/appbase-js/blob/develop/src/core/index.js#L51

Improve error handling

Currently errors are emitted on the streams response, and appbaseio/reactivecore:src/actions/index.js@master#L150-L153 handleError() is never getting called. Check appbase-js related methods for this.

Ref - appbaseio/reactivesearch#163