deepstreamio / deepstream.io-storage-mongodb

A deepstream.io storage connector for MongoDb

deepstream.io-storage-mongodb

deepstream storage connector for mongodb

This connector uses the npm mongodb package. Please have a look there for detailed options.

Basic Setup

plugins:
  storage:
    name: mongodb
    options:
      connectionString: ${MONGODB_CONNECTION_STRING}/someDb
      defaultCollection: 'someTable'
      splitChar: '/'

var Deepstream = require( 'deepstream.io' ),
    MongoDBStorageConnector = require( 'deepstream.io-storage-mongodb' ),
    server = new Deepstream();

server.set( 'storage', new MongoDBStorageConnector( {
  connectionString: 'mongodb://test:[email protected]:10087/munchkin-dev',
  splitChar: '/'
}));

server.start();

deepstream.io-storage-mongodb's Issues

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.

ds_key index on mongo collections are not unique

Is it by design that ds_key are not unique indexes on mongodb?

 this._collections[ collectionName ].ensureIndex({ ds_key: 1 })

Since the value cannot be duplicated, should it not be more optimized to use?

 this._collections[ collectionName ].ensureIndex({ ds_key: 1 }, {uniq:1})

Would it not be more optimized to use ds_key as as the _id of the record?
(Just asking what the design considerations were?)

ds_key index is not created

This gives an error and the index is not created:
this._collections[ collectionName ].createIndexes({ ds_key: 1 });

Fix it:
this._collections[ collectionName ].createIndex({ ds_key: 1 });

Using multiple nested keys throws error (#9), found a fix which works, but your tests have failed. Any side effects expected ?

( Apologies, Wasn't sure how to re-open or request re-opening of a previously closed issue (#9), so creating a new one. New to the github process. Reposting below )

Faced the same issue as in #9 recently, despite being closed. Have fixed it at my end and submitted pull request #11 and it seems to be working correctly for now, but your tests on the pull request have failed. Does that imply any side effects ? Please advise !

using
mongodb storage connector 1.1.0
deepstream 2.3.6

on Digital Ocean, configured according to the official Deepstream Digital ocean tutorial using deepstream + redis + mongodb.

the pull request has failed checks with the message - " expected 'a/b' to equal 'a' "

Update config options docs

database is part of the connection string,
defaultTable is called defaultCollection

storage:
    name: mongodb
    options:
      connectionString: 'mongodb://localhost:27017/someDb'
      defaultCollection: 'someTable'
      splitChar: '/'

PLUGIN_INITIALIZATION_ERROR | storage needs to implement async whenReady and close, please look at the DeepstreamPlugin API here

const { Deepstream } = require('@deepstream/server')
const deepstream = require('@deepstream/client')

const MongoDBStorageConnector = require('deepstream.io-storage-mongodb')

const server = new Deepstream({})

server.set('storage', new MongoDBStorageConnector({
  connectionString: 'mongodb://localhost:27017/db',
  splitChar: '/'
}))

server.start()

"@deepstream/server": "^4.0.6"
"deepstream.io-storage-mongodb": "^1.1.1"

_getParam fails for keys with multiple splitchars

according to this tutorial:
https://deepstream.io/tutorials/core/permission-conf-simple/
```record:
   auction/item/$sellerId/$itemId:```
(edited)

[9:56]  
in a Record I can set the collection name, a key, and many custom parameters

[9:56]  
but when I try to do a similar form

[9:57]  
demo/586bcdc0c7a2fd181d000006/abc/123

[9:57]  
it gives me this error: from storage:Invalid key

This can be solved by changing https://github.com/deepstreamIO/deepstream.io-storage-mongodb/blob/master/src/connector.js#L188 to only split it on the first splitChar

Pagination list records ?

How can i do pagination list records . ? . Example record.list('chats/*', { start: 0, limit: 256 }, ({ ids }) => console.log(ids))

Website connectionString documentation out of date

The Website documentation here:
https://deepstreamhub.com/open-source/integrations/db-mongodb/
Shows the "older" format for mongodb connection in the conf.yaml for deepstream

storage:
  name: mongodb
  options:
    connectionString: ${MONGODB_CONNECTION_STRING}
    # optional database name, defaults to `deepstream`
    database: 'someDb'
    # optional table name for records without a splitChar
    # defaults to deepstream_records
    defaultTable: 'someTable'
    # optional character that's used as part of the
    # record names to split it into a tabel and an id part
    splitChar: '/'

where is should be according to this repo:

storage:
  name: mongodb
  options:
    connectionString: ${MONGODB_CONNECTION_STRING}/somedb
    # DEPRECATED? optional database name, defaults to `deepstream`
   #  DEPRECATED? database: 'someDb' --- NOT used in newer versions of mongo
    # optional table name for records without a splitChar
    # defaults to deepstream_records
    defaultTable: 'someTable'
    # optional character that's used as part of the
    # record names to split it into a tabel and an id part
    splitChar: '/'

Using multiple nested keys throws error

Issue

Using a key like x/y/z presents an error when using mongodb as the storage back-end. Using a key like x/y works as intended. Basically any key with more than one nesting throws error.

Environment

Latest docker
Latest image pulled from deepstream/deepstream.io
Latest mongoDb image from DockerHub

Example client code:
"const record = deepstreamClient.record.getRecord('settings/user/hi')"

Example server config:
Stock config except mongodb backend:
storage:
name: mongodb
options:
connectionString: 'mongodb://mongodb:27017'
database: 'deepstream'
defaultTable: 'toplevel'
splitChar: '/'

Server error:
RECORD_LOAD_ERROR | error while loading settings/user/hi from storage:Invalid key settings/user/hi

The field 'indexes' must be an array, but got object

Trying to use V4 server and client with @deepstream/storage-mongodb@v4 and got this error:

(node:9297) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
{ MongoError: The field 'indexes' must be an array, but got object
    at Connection.<anonymous> (/home/timar/prog/moc/ds-server/node_modules/mongodb-core/lib/connection/pool.js:443:61)
    at Connection.emit (events.js:198:13)
    at processMessage (/home/timar/prog/moc/ds-server/node_modules/mongodb-core/lib/connection/connection.js:364:10)
    at Socket.<anonymous> (/home/timar/prog/moc/ds-server/node_modules/mongodb-core/lib/connection/connection.js:533:15)
    at Socket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
  ok: 0,
  errmsg: 'The field \'indexes\' must be an array, but got object',
  code: 14,
  codeName: 'TypeMismatch',
  name: 'MongoError',
  [Symbol(mongoErrorContextSymbol)]: {} }
TypeError: Cannot set property '__ds' of undefined
    at Object.module.exports.transformValueForStorage (/home/timar/prog/moc/ds-server/node_modules/@deepstream/storage-mongodb/src/transform-data.js:28:15)
    at DSStorage.Connector.set (/home/timar/prog/moc/ds-server/node_modules/@deepstream/storage-mongodb/src/connector.js:110:25)
    at DSStorage.storage.set (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/config/config-initialiser.ts:362:12)
    at RecordHandler.create (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/handlers/record/record-handler.ts:425:29)
    at RecordHandler.onPermissionResponse (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/handlers/record/record-handler.ts:645:7)
    at RuleApplication.run (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/services/permission/valve/rule-application.ts:102:19)
    at new RuleApplication (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/services/permission/valve/rule-application.ts:64:10)
    at DSPermission.canPerformAction (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/services/permission/valve/config-permission.ts:142:5)
    at RecordHandler.permissionAction (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/handlers/record/record-handler.ts:605:30)
    at RecordHandler.onCreateOrReadComplete (/home/timar/prog/moc/ds-server/node_modules/@deepstream/server/src/handlers/record/record-handler.ts:277:12)

Deepstream server setup using mongo connector:

const MongoDBStorageConnector = require('@deepstream/storage-mongodb');

const server = new Deepstream({/*...*/});

server.set('storage', new MongoDBStorageConnector({
        connectionString: "mongodb://username:[email protected]:27017/ds-records?retryWrites=true&authSource=ds-records",
        splitChar: '/',
      }));

server.start();

@deepstream/server: ^4.0.0-rc.17
@deepstream/client: ^4.0.0-rc.42
@deepstream/storage-mongodb: ^2.0.1
@deepstream/cache-redis: ^2.0.0-rc.12

node -v
v10.16.0

Duplicate record for new collections/tables

{"_id":"ObjectId()","text":"something","__ds":{"_v":1},"ds_key":"ixqb796a-1ezjx1aff6m"}
{"_id":"ObjectId()","__ds":{"_v":0},"ds_key":"ixqb796a-1ezjx1aff6m"}

when I create a new Record for a Collection that was not existed before it also creates a duplicated document with the same ds_key what the first Record will have with a version 0
it is not a big problem as I can avoid finding it but does it have a purpose or am I doing something wrong with how I set my first Record?
this only happens with the very first Record for an empty Collection only, after that no similar problems arise

Update MongoDB driver to v3

The new MongoDB driver has breaking API changes.