toystars / node-elasticsearch-sync Goto Github PK

I've had an issue inserting documents using a BSON id straight from MongoDB. If found a solution, but wonder if there is a particular reason why here

https://github.com/toystars/node-elasticsearch-sync/blob/master/lib/util/crud.js#L39

document.id is used "as-is" in the insert function, while in the remove and update function, id.toString() is used, see

https://github.com/toystars/node-elasticsearch-sync/blob/master/lib/util/crud.js#L62
https://github.com/toystars/node-elasticsearch-sync/blob/master/lib/util/crud.js#L86

Wouldn't it be OK to use id: document.id.toString() in the insert function as well?

Hello ,
Mongo information : DB name : fortitude , collection name : nodes , number of documents : 19
ES cluster information : http://ip:9200/

This is my main.js : ( main.js )

var ESMongoSync = require('node-elasticsearch-sync');

let transformFunction = (watcher, document, callBack) => {
document.name = document.firstName + ' ' + document.lastName;
callBack(document);
}

let sampleWatcher = {
collectionName: 'nodes',
index: 'fortitude',
type: 'nodes',
transformFunction: null, // can be null
fetchExistingDocuments: true,
priority: 0
};

// the "collectionName" and "type" fields in watchers MUST be the same. This might change in later versions.

let watcherArray = [];
watcherArray.push(sampleWatcher);

// The following env_vars are to be defined. Error will be thrown if any of the env_var is not defined

ESMongoSync.init(watcherArray, null);

This is the output of my code ,

[email protected]:/opt/node_modules/node-elasticsearch-sync #node main.js

ESMongoSync: Connected to ElasticSearch successfully!
ESMongoSync: Connected to MONGO server successfully.
ESMongoSync: Beginning batch creation
ESMongoSync: Processing watchers on priority level 0
ESMongoSync: Processing nodes collection
ESMongoSync: Batch creation complete. Processing...
ESMongoSync: Number of documents in batch - 100
ESMongoSync: Batch processing complete!
ESMongoSync: Oplog tailing connection successful.

Nothing is written in the Elastic Search .

I donot find about GridFS function in your code. How to sync fridfs? Thanks

When I insert a new Document in mongodb,the mongo-oplog return objectId is messy code in the nodeCode.I use vsCode debug this.

HI This is great App.

I have it working for inserting data in mongo and it syncs to Elasticsearch, however in MongoData there is a _id field which it didnt like so i

addeddocument.MongoID = document._id;
document = _.omit(document, '_id');

to the transformFunction, which was ok, but when deleting records from Mongo, it gives error

/Users/collia01/Node/rabbittomongo/node_modules/node-elasticsearch-sync/lib/util.js:143
if (!response.found) {
^

TypeError: Cannot read property 'found' of undefined
at /Users/collia01/Node/rabbittomongo/node_modules/node-elasticsearch-sync/lib/util.js:143:20
at Object.utils.applyArgs (/Users/collia01/Node/rabbittomongo/node_modules/elasticsearch/src/lib/utils.js:252:17)
at wrapper (/Users/collia01/Node/rabbittomongo/node_modules/elasticsearch/node_modules/lodash/index.js:3533:19)
at nextTickCallbackWith0Args (node.js:433:9)
at process._tickCallback (node.js:362:13)

Is this because the _id has been renamed in elasticseach? Is there a way around this.

Thank you
Andy

I have a json array in memory. It needs to be synced to elastic search. Can you add support ?

I have an issue with collection name != type name, and I think it is down to this line

https://github.com/toystars/node-elasticsearch-sync/blob/master/index.js#L64

Should that not do something like currentCollectionName = document.collection ?

Hello, I am facing this problem and I am not understanding why. The actual code are in this state:

let transformFunction = (watcher, document, callBack) => {
  //document.name = document.firstName + ' ' + document.lastName;
  //console.log(document);
  callBack(document);
}

let componentsWatcher = {
  collectionName: 'components',
  index: 'meteor.components',
  type: 'components',
  transformFunction: transformFunction,
  fetchExistingDocuments: true,
  priority: 0
};

And it is giving me the currently error, which stop from synchronising.
Any idea ? Thank you

When i run

ESMongoSync.init("mongodb://localhost:27017/db","localhost:9200", finalCallBack, watcherArray, batchCount);

i get this error
throw new TypeError("Parameter 'url' must be a string, not " + typeof url);

The documents in my MongoDB schema have an id attribute, which is overwritten by the default transform(..) function. I see different options to make this more flexible (e.g. with an environment variable or by re-introducing settings), but the simplest solution seems to be to only make the "id shift" if the ID parameter does not exist, log a warning, and let the user handle the copying of the id in their transform function.

I am adding changes to that effect to #14

@nuest I expected to see your name as part of the contributors list in the README.

ESMongoSync: Oplog tailing connection successful.

<--- Last few GCs --->

6182 ms: Scavenge 1412.6 (1457.8) -> 1412.6 (1457.8) MB, 4.5 / 0 ms (+ 6.4 ms in 1 steps since last GC) [allocation failure] [incremental marking delaying mark-sweep].
6360 ms: Mark-sweep 1412.6 (1457.8) -> 1412.6 (1457.8) MB, 178.4 / 0 ms (+ 21.1 ms in 7 steps since start of marking, biggest step 6.4 ms) [last resort gc].
6544 ms: Mark-sweep 1412.6 (1457.8) -> 1412.6 (1457.8) MB, 184.5 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2aecf56e3ac1
1: /* anonymous /(aka / anonymous */) [/home/abhilash.bolla/Learning/elasticsearch_query_example/node_modules/mongodb/node_modules/bson/lib/bson/parser/deserializer.js:~39] [pc=0x1c5ae0e609d2](this=0x2aecf5604189 ,buffer=0x25ef6ed1ec91 <an Uint8Array with map 0x36a53f052e1>,index=2393118,options=0x25ef6ed1ece1 <an Object with map 0x36a53f1b549>,isArray=0x2aecf5604231 <true)
...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
Aborted (core dumped)

Getting this error.

We want to run this in its own container and do the syncing.
I was wondering if running multiple instances of this watchers mean that all of them will always rebuild ES index.

Hi there. Very useful package you have there! Would you be interested in a contribution with extended logging statements using the debug module (see https://www.npmjs.com/package/debug)? I've had to hunt down an issue in my usage of the code and some extended configurable logging would have helped.

I am calling the function as :

ESMongoSync.init(null, null, null, watchers, 500);

Please assist if its wrong.

Currently the package has no test - but it is really useful to me, so I'll add this "request" knowing that I'll probably have to do this myself :-)

Since this package really needs an actual MongoDB and Elasticsearch to be tested, integration tests could run both of these databases in Docker containers and then insert data into the MongoDB and check that they are available in Elasticsearch.

Here are some links that could get somebody started on this task:

• use compose and a script (start the containers "around" the tests) https://hharnisc.github.io/2016/06/19/integration-testing-with-docker-compose.html
• use dockerode (start the containers from within the script) https://www.npmjs.com/package/dockerode