sergeyksv / tingodb Goto Github PK

View Code? Open in Web Editor NEW

1.2K 1.2K 106.0 4.94 MB

Embedded Node.js database upward compatible with MongoDB

Home Page: www.tingodb.com

JavaScript 99.93% Shell 0.07%

tingodb's People

Contributors

Stargazers

Watchers

Forkers

awwright rmechler rakesh-mohanta lahdekorpi brettveenstra pwave hihihippp al-s node4good dist speefictor jugalkishor-sujal mykkro juniperchicago rd-wixproducts unclegigi nicolasritouet firdausramlan themoooooooooo mutac rezaf79 albatrosary elasim student007 jubolin mefozzy frederik casualuser bipsync muka sealcode angelochen960 robertomalatesta bb-costik luphia andrethehunter zen-li valenti1234 jonasho okusnadi egreen-lab jdrew1303 bussiere itmgr john-coffey comker rootfs-analytics mamuuut diogoduailibe bg1avd sandorlevi shunr javascriptiot pwd3 lmssrinivas rlkelly thomasderzweifler farfromrefug antonpuko brewskey cnuser ex1st besworks weimingtom kafkaforks ndastur rokikon eduel ramy-ahmed root-core apowers313 roxberry solorzanof2 tocilla anhoev lieutenantroger alanning pks-os random-developer cedececa lcnvdl ikbhal ilanvolow dewoodruff kimthiendung gitterebi tluyben yunwisdom xdata-database-repository octopoulos pasoon alst-ovh vishnuh655 sevenbitbyte gaybro8777 uboness szf2020 alkbur cfcomputing gmh5225

tingodb's Issues

Does Mongodb implement "distinct" mechanism

Hi,

I've started to use your excellent library and I'm facing an issue concerning retrieving a set of unique results from a database. Currently I'm achieving a reduce operation after having retrieved all data to fix out the problem but I'll prefer to delegate this filtering task to the database engine; Is it possible to do so with the current tingodb implementation?

Thanks in advance.

tcoll.prototype._compact does not work on windows

I am relatively new to node.js and TingoDB, so please forgive my ignorance if I am incorrect in my assessment here.

When restarting my software which uses TingoDB, an attempt is made to rename the "spec.compact" file to the "spec" file (replacing one by the other). The problem is that the "spec" file exists and is open, so the operation fails with the following error:

{ [Error: EPERM, rename 'C:\Users\Gern Blanston\Documents\twoducks\dal\git.qvtrace\qvtrace\webservice\db\specs.compact']
  errno: 50,
  code: 'EPERM',
  path: 'C:\\Users\\Gern Blanston\\Documents\\twoducks\\dal\\git.qvtrace\\qvtrace\\webservice\\db\\specs.compact' }

I updated the tcoll.prototype._compact function to ensure the file is closed and deleted BEFORE running and everything appears to operate normally:

...
            // Was not properly renaming on windows
            fs.close(self._fd, function() {
                fs.unlink(self._filename, function () {
                    fs.rename(filename, self._filename, safe.sure(cb, function () {
                        //fs.close(self._fd); // Already closed
                        self._fd = fd;
                        self._fsize = wpos;
                        self._store = store;
                        cb();
                    }));
                });
            });

Support $push for update?

Any plans to support this? Right now it just fails silently (i.e., no error, but no modification is made).

Considerations for a possible usecase

Hi there,
I have an old IRC bot written in Perl and combined with SQLite. It has been running for a few years and has accumulated several gigabytes of channel messages that I would like to be accessible. My perl bot allows for arbitrary regexp searches (slow but works) and I read that Mongo would optimize some of the regexes in order to utilize the index. I love MongoDB for other projects but I despise the approach of utilizing some external database server and hence TingoDB - this project looks awesome? Would it work great in my usecase? Of course the bot has other features but I am mostly concerned about sipping through the gigabytes of channel messages.
I look forward to your reply.

Collection update results in duplicates

Result of following code is two entries in db file.

var Db = require('tingodb')().Db;
var db = new Db('lastexcuse.db', {});
var cb = function (err, result) {
    console.log(err, result);
};
db.collection('accounts').insert({id: 5, name: 'majo'}, cb)
db.collection('accounts').update({id: 5}, {name: 'fero'}, cb)

encryption/authentication of database

I dont really know if this is possible in tingodb, but is there a way to encrypt the database and grant access just to some users to use it?.
MongoDB has.
Having local stored data accesible just usint a text editor would violate the integrity of the app and the information that is working with.

Support in memory usage.

It would be nice if this could work without need to specify a filesystem path. It would make it easier for unit tests, since they wouldn't need to clean up.

Errors are silenced

Take this simple example script:

var Db = require('tingodb')().Db;
var db = new Db('/tmp/tests', {});
var collection = db.collection( 'addresses' );

a = {
  personId: 555,
  street  : 'bitton',
  city    : 'perth',
  id      : 12345
};    


collection.remove( {}, function( err ){
  if( err ) return process.exit( err );

  collection.insert( a, function( err ){

    if( err ) return process.exit( err );

    collection.update( { '$and': [ { id: 12345 } ] }, { street: 'bitton CHANGED', __uc__street: 'BITTON CHANGED' }, function( err, n ){

      if( err ) return process.exit( err );
      console.log("CHANGED:", n );

      collection.find({}).toArray(function( err, res ){

        if( err ) return process.exit( err );

        console.log( res.iDoNotExist() );
        console.log("DATA:", res  );
      });

    });
  });
});

res.iDoNotExist() obviously does not exist. And yet, there is no error message -- the error is quite silenced, and the script dies.

It should throw an exception... right?

cursor.count() resets cursor if there is a search filter in the query

The comment says it all. The TingoTicket.js script explains it in detail!
Basically, cursor.count() should never reset the cursor.

var 
  dummy
, Db = require('tingodb')().Db;
;


var db = new Db('/tmp/tests', {});
var c = db.collection( 'test' );


function printErr( err ){
  if( err ){
    console.log("ERROR!");
    console.log( err );
  }
}

function populate( cb ){

  console.log("Zapping and populating the DB...");
  c.remove( {}, function( err ){
    c.insert( {  name: 'Chiara',    surname: 'Mobily',     age: 22 } , function( err ){
      c.insert( {  name: 'Tony',    surname: 'Mobily',     age: 37 } , function( err ){
        c.insert( {  name: 'Sara',    surname: 'Connor',     age: 37 } , function( err ){
          c.insert( {  name: 'Daniela',    surname: 'Mobily',     age: 37 } , function( err ){
            cb( err );
          })
        })
      })
    })
  })
}

function idProjection( cb ){

  c.find( {}, { name: 1, age: 1, _id: -1 }, function( err, res ){
    if( err ){
      cb( err );
    } else { 
      res.toArray( cb );
    }

  })  
}


function countResetsCursorWorks( cb ){

  cursor = c.find(  { }, { }  );

  cursor.count( function( err, total ){
    if( err ){
      cb( err );
    } else {
      console.log("TOTAL:");
      console.log( total );

      cursor.toArray( function( err, total ){
        if( err ){
          cb( err );
        } else { 
          cursor.count( cb );
        }

      })
    }
  })
}

function countResetsCursorDoesNotWork( cb ){

  //cursor = c.find(  { '$and': [ { name: 'Tony' }, { surname: 'Mobily' }, { age: 37 } ] }, { name: true, surname: true, age: true }   );
  cursor = c.find(  { name: 'Tony', surname: 'Mobily',  age: 37 }, { name: true, surname: true, age: true }   );

  cursor.count( function( err, total ){
    if( err ){
      cb( err );
    } else {
      console.log("RECORDS COUNTED:");
      console.log( total );

      cursor.toArray( function( err, total ){
        if( err ){
          cb( err );
        } else { 
          cursor.count( cb );
        }

      })
    }
  })
}


function sortOnlyOneField( cb ){
  c.find( {}, { sort: { surname: 1, name: 1 }  }, function( err, results ){
    if( err ){
      cb( err );
    } else {
       results.toArray( function( err, a ) {
         console.log("RESULTS SHOULD BE ORDERED BY SURNAME,NAME:");
         console.log( a );
         console.log();
         cb( null );
       });
    }
  });
}

function regexpBroken( cb ){
  // c.find( { '$and': [ { surname: { '$regex': /.*nor$/ } } ] } , function( err, results ){
  c.find( { surname: { '$regex': /.*nor$/ } }, function( err, results ){
    if( err ){
      cb( err );
    } else {
       results.toArray( function( err, a ) {
         console.log("RESULTS SHOULD BE LIMITED:");
         console.log( a );
         console.log();
         cb( null );
       });
    }
  });
}




populate( function( err ){

  idProjection( function( err ){
    printErr( err );

    countResetsCursorWorks( function( err ){
      printErr( err );

      countResetsCursorDoesNotWork( function( err ){
        printErr( err );

        sortOnlyOneField( function( err ){
          printErr( err );

          regexpBroken( function( err ){
             printErr( err );
             db.close();
          })   
        }); 
      });
    });
  });
})

tingodb and concurrency

Hi,

Does TingoDB manage concurrent requests or does I have to manage it myself with a semaphore ?
I've encountered some strange behaviours when I tried to update the same document in two different callbacks. In this case, the final document didn't contain modifications made by one of the two callbacks.

Thanks in advance.

openCalled variable in tdb.js

MongoDB uses openCalled variable to inform that an open or close was called on db. Some modules, like connect-mongo, refer to this variable.

Please implement { applySkipLimit: true }, function( err, total ){

Implement positional operator $ for array updates

The title says it all...

I have recently expanded simpledblayer (which has tingoDb as a possible backend) so that it actually simulates joins. The way it works is quite advanced: each record keeps an updated list of its children; when a record is updated, simpledblayer also update any parent record that includes the modified one as a child (!).

Trouble is, my tests are failing (11 out of 287) because updating with positional operator doesn't work.

I had a look at the code, and couldn't make heads of tails of it. But, I would love to have that feature there...

Thank you!

Undefined hint

The following backtrace is occurring in the findAndRemove function, which was just called with (query, cb):

/home/vagrant/p4/vwf/node_modules/tingodb/lib/tcoll.js:725
                self.__find(query,null,0,1,c._sort,c._order, opts.hint, {}, safe.sure(cb, fu
                                                                 ^
TypeError: Cannot read property 'hint' of undefined
    at /home/vagrant/p4/vwf/node_modules/tingodb/lib/tcoll.js:725:52
    at wqueue._exec (/home/vagrant/p4/vwf/node_modules/tingodb/lib/wqueue.js:36:3)
    at /home/vagrant/p4/vwf/node_modules/tingodb/lib/wqueue.js:54:10
    at process._tickCallback (node.js:415:13)

Here is the mongodb driver. It looks like it treats sort and options as optional. When I was calling findAndRemove above, it was just calling it with a query and callback. Tingo expects four parameter... certainly doesn't treat the second of two parameters as the callback.

Here is the mongo driver implementation. Notice how they pop the callback off the end.

Collection.prototype.findAndRemove = function(query, sort, options, callback) {
  var args = Array.prototype.slice.call(arguments, 1);
  callback = args.pop();
  sort = args.length ? args.shift() || [] : [];
  options = args.length ? args.shift() || {} : {};
  // Add the remove option
  options['remove'] = true;
  // Execute the callback
  this.findAndModify(query, sort, null, options, callback);
}

remove: no error & returns 0, but record still there. Further .remove bugs too.

I'm updating an object, in callback checking if its id needs to be 'updated' also, if yes, duplicate obj with new id & insert. Then .remove old object.

.remove works fine when alone, but after the ^ above, even my normal .remove starts behaving the above way.

All this is done with the same collection instance.

Getting a TypeError: Object false has no method 'test' when searching with RegExp

If you have this data:

{"k":"0000000082","o":"0000000123","v":"001"}
{"_id":642,"_uid":645,"_dt":1384988597803,"_s":"94525e8e220a58775102a36da1bc6542"}
{"name":"Chiara","surname":"Tobily","age":22,"_id":{"$wrap":"$oid","v":642},"__uc__name":"CHIARA","__uc__surname":"TOBILY"}
{"k":"0000000082","o":"0000000096","v":"001"}
{"_id":643,"_uid":645,"_dt":1384988597806,"_s":"e148f298712c3087d8005afdfa3e46a2"}
{"surname":"Sobily","_id":{"$wrap":"$oid","v":643},"__uc__name":"TONY","__uc__surname":"SOBILY"}
{"k":"0000000082","o":"0000000119","v":"001"}
{"_id":644,"_uid":644,"_dt":1384988597800,"_s":"ec7e4264ec8672ed7cd7b6eddba81b3e"}
{"name":"Sara","surname":"Connor","age":14,"_id":{"$wrap":"$oid","v":644},"__uc__name":"SARA","__uc__surname":"CONNOR"}
{"k":"0000000082","o":"0000000125","v":"001"}
{"_id":645,"_uid":645,"_dt":1384988597803,"_s":"b2d3a9a963bb7bc072a5cda519fd7cce"}
{"name":"Daniela","surname":"Tobily","age":64,"_id":    {"$wrap":"$oid","v":645},"__uc__name":"DANIELA","__uc__surname":"TOBILY"}

And try this script:

var Db = require('tingodb')().Db;

// Catching everything that bubbles up uncaught
process.on('uncaughtException', function(err) {
  console.error( err.stack );
});

var db = new Db('/tmp/tests', {});

var collection = db.collection( 'people' );

cursor = collection.find( { '$and': [ { __uc__surname: { '$regex': /.*ON.*/ } } ] }, { name: true, surname: true, age: true } );

cursor.sort( {} , function( err ){

  cursor.toArray( function( err, queryDocs ){
    if( err ){
     console.log("ERROR AFTER toArray!!!");
     console.log( err );
     console.log( err.stack );
    } else {
      console.log("QueryDocs:");
      console.log( queryDocs );

      cursor.count( { applySkipLimit: true }, function( err, total ){
        if( err ){
          console.log("ERROR AFTER count!!!");
          console.log( err );
        } else {
          console.log("Total: " + total );
        }
      });
    }

  });
});

You will get:

$ node TingoTicket3.js
ERROR AFTER toArray!!!
[TypeError: Object false has no method 'test']
TypeError: Object false has no method 'test'
    at matcher (eval at <anonymous> (/home/merc/Synced/Development/node/node_modules/tingodb/lib/tcoll.js:829:83), <anonymous>:1:110)
    at Object.<anonymous> (/home/merc/Synced/Development/node/node_modules/tingodb/lib/tcoll.js:841:8)
    at Object.<anonymous> (/home/merc/Synced/Development/node/node_modules/tingodb/node_modules/safe/lib/safe.js:31:7)
    at Object._onImmediate (/home/merc/Synced/Development/node/node_modules/tingodb/node_modules/safe/lib/safe.js:83:6)
   at processImmediate [as _immediateCallback] (timers.js:330:15)
merc@daniela-ThinkPad-Edge-E320:~/Synced/Development/node/dev/node_modules/simpledblayer-tingo$

Поиск по тексту

Отчаялся искать решение, решил спросить.

Есть коллекция:

{"id":1,"first_name":"Alexandr","last_name":"Pri"}
{"id":2,"first_name":"Alex","last_name":"Nub"}
{"id":3,"first_name":"Alena"}
{"id":4,"last_name":"Kuznecev"}

И есть поле поиска в которое пользователь вводит ЧАСТЬ имени контакта, например пользователь ввел "Alex", как найти в контакты с ID 1 и 2?

Если полнотекстовый поиск не реализован в tingodb, возможно вы подскажите другое решение как выполнять такой поиск?

cursor.sort() если не везде есть поля

В БД есть данные:

{"id":1,"first_name":"Liza","last_name":"Pri"}
{"id":2,"first_name":"Alex","last_name":"Nub"}
{"id":3,"first_name":"Vanya"}
{"id":4,"last_name":"Kuznecev"}

Если попытаться отсортировать коллекцию

collection.find().sort({first_name: 1});

в консоль "капнет" ошибка синтаксиса.
Если же у контактов с id 3 и 4 в коллекции, добавить фамилию и имя - ошибка исчезает и все сортируется.

P.S.: Допускаю что я что-то не понимаю, если не сложно объясните как сортировать такие коллекции?

How to use TingoDB inside of node-webkit?

I'd like to use TingoDB inside of node-webkit, and the website says that this should work:

TingoDB is embedded JavaScript NoSql database for Node.js and node-webkit.

Anyway, I could not find any example of how to set it up. It's obvious that you need to install the npm module, require it and so on … my question basically is what to insert as database path in line:

var tingo = require('tingodb')();
    db = new tingo.Db('/some/local/path', {});

What's the node-webkit equivalent to /some/local/path?

.compact files error

Sometimes, when I call collection.remove(), there will be an error:

[Error: EPERM, rename '<collection>.compact']
errno: 50,
code: 'EPERM',
path: '<collection>.compact' }]

but the function works actually.

So, I want to know why and when does the .compact file generate?
How to kill the error?

Did you run npm publish?

I wonder if you have run npm publish with the latest fixes.
The current version in npm is 0.1.15, whereas on GIT it's 0.1.17.

Thanks!

$unset assigns 'undefined' rather than deleting a record

This is not a biggie, but it's there.
Imagine you have a collection with only one document, containing [ { name: 'Tony', age: '37' } ].
At the moment, writing:

self.collection.findAndModify( {}, {}, { $set: {name: 'Tony'}, $unset: { age: true } }, function( err, doc ){

Will modify the doc like this: { name: 'Tony', age: undefined } which is not what mongo does (it sets the doc as { name: 'Tony' }

Thanks for listening!

ObjectID Ignored on Insert

id = new db.ObjectID();
// in console, id: ObjectID {id: -586, _persist: function, _init: function, inspect: function, toString: function…}

collection.insert({'_id' : id, data: 'test'});

This should insert a document with the id of -586? However, when I loop through the data, the new document has an id of 138 just like if I didn't specify an id at all.

Не работает update при использовании массива

Если поле содержит массив его нельзя перезаписать.
Например есть данные:

{
    "id":3044794,
    "first_name":"Jon",
    "middle_name":"Micke",
    "last_name":"Smit",
    "phone": [
        {"type": "home", "number": "956-18-23"},
        {"type": "work", "number": "322-52-22"},
        {"type": "other", "number": "232-23-22"}
    ]
}

При попытке заменить массив номеров на другой, возникает ошибка: "Object 0 has no method 'split'"

collection.update({_id: id}, {$set: {"phone": [{"type": "mobile", "number": "3323413"}, {"type": "other", "number": "4423"}]}});

Допускаю что я что то не так делаю.
Спасибо.

How to do fast queries on big databases ?

I have a huge database, with 200k+ entries, and would like to know if there is a fast way of querying it?

I will be mainly doing "search" functions on it, like "filtering" by values in a given column that matches a regular expression.

Is there a recommended way of dealing with such big db ?

value in finder.js logs warning for valid value

@ line 749 of finder.js, which was
else
console.log("WARNING: Using unrecognized value in query",v)

should instead read
else if (this.v !== null)
console.log("WARNING: Using unrecognized value in query",v)

Alternately, this.v === null could be added as an or for the first if condition.

Passing a value as null does work, and is required for some circumstances, particularly searches for unset values. Previously doing so caused log messages warning of the "unrecognized" value.

Query with object as value didn't work same way as with Mongo

OBJECT:
{ "_id" : { "$oid" : "52307e98f95a501244005feb"} , "cmdty" : { "space" : "ISO4217" , "id" : "EUR"} , "currency" : { "space" : "ISO4217" , "id" : "RUB"} , "date" : { "$date" : "2010-03-03T21:00:00.000Z"} , "source" : "transaction" , "value" : 40.648936170212764}

QUERY:
{cmdty: { space: 'ISO4217', id: 'RUB' }, currency: { space: 'ISO4217', id: 'USD' } }

Works in Mongo, didn't work in Tingo. However to be hones I never though it can work with Mongo as well.

Aggregate Framework Support

Aggregation support would be extremely awesome. 👍 Any plans to include it on the roadmap?

findAndModify() needs to return the updated record, not the old one

The call findAndModify() needs to return the actual record, rather than the version of the record before modifications.
This simple script shows exactly the problem:

var Db = require('tingodb')().Db;
var db = new Db('/tmp/tests', {});
var collection = db.collection( 'addresses' );

a = {
  personId: 555,
  street  : 'bitton',
  city    : 'perth',
  id      : 12345
};

console.log("Removing...");
collection.remove( {}, function( err ){
  if( err ) return process.exit( err );

  console.log("Inserting...");
  collection.insert( a, function( err ){

    if( err ) return process.exit( err );

    console.log("Finding...");
    collection.findAndModify( { '$and': [ { id: 12345 } ] }, {}, { $set: { street: 'bitton CHANGED', __uc__street: 'BITTON CHANGED' }, $unset: {} } , function( err, d ){

      console.log( err );
      if( err ) return process.exit( err );

      console.log("RETURNED:", d );

      collection.find( {} ).toArray( function( err, res ){
        if( err ) return process.exit( err );

        console.log("Results:", res);

      });
    });
  });
});

If data directory doesn't exist, when insert()ing the callback isn't called at all

Consider the following code:

var Db = require('tingodb')().Db;

// Catching everything that bubbles up uncaught
process.on('uncaughtException', function(err) {
  console.error(err.stack);
});


// NOTE: MAKE SURE THAT /tmp/test DOESN'T EXIST
var db = new Db('/tmp/tests', {});

var c = db.collection( 'test' );

c.remove( {}, function( err ){
  console.log("I AM HERE. OK, now let's go on...");
  c.insert( {  name: 'Chiara',    surname: 'Mobily',     age: 22 } , function( err ){
    console.log("THIS WILL NEVER DISPLAY");
  })
})

If /tmp/test doesn't exist, the second callback is never called. No error is thrown (it would be caught by uncaughtException), and the callback isn't called at all -- not even with err set. Everything just fails quietly.

I actually tried to fix it, but got lost in the code -- sorry.

Querying cursor results with "find" method ?

I would like to do a "find" query, and then executing another "find" on the resulting cursor, therefore "filtering" even more the results.

Is that possible somehow ?

Sometimes, safe option is not safe!

I insert some docs though the following code

collection.insert(docs, {w: 1, safe: true}, function(err2, resultDocs){
      if (err2) {
        logger.error(err2);
        if(cb){
          cb(err2, []);
        }
      }else{
        if(cb){
          resultDocs.forEach(function(doc){
            doc.id = doc._id.id;
          });
          cb(null, resultDocs);  
        }
      }      
    });

From my log, I am sure that the call back function is called and there is no error. But when I query the table, the data is not inserted. I open the table file to have a check, and find that: the rows that is not inserted have the same "_uid" with other rows. This is a data snippet:

{"_id":2,"_uid":2,"_dt":1404184134069,"_s":"f81690a4c75b8eef254960be7f5915e6"}
{"name":"DemoTheme","label":"Demo Theme","panels":[{"name":"SimpleBorderPanel",......
{"k":"0000000078","o":"0000001412","v":"001"}
{"_id":2,"_uid":2,"_dt":1404184134092,"_s":"e86292c2907529c417976e89c9fbaffc"}
{"name":"FoldableTheme","label":"Foldable Theme","panels":[{"name":"FoldablePanel,......

ISODate support

TingoDB is life saver for embedded apps, running it on PC and Raspberry PI
When using ISODate() I get ReferenceError: ISODate is not defined does TingoDB support this helper.

A big thanks for your great work.
Lawrence

Compacting failure

Under some circumstances during compacting, renaming collectionfile.compact fails. This leaves an empty collectionfile, giving the user the impression that all data was suddenly lost. Closing nodejs, manually renaming collectionfile.compact to collectionfile and restarting nodejs seems to be a functioning workaround. I don't know how to force a compact, so I cannot provide steps to reproduce.

Versions:
tingodb 0.2.1
nodewebkit 0.8.5
npm 1.4.3 ("npm start" launched from a cygwin shell)
node v0.10.26 (pre-built executable, non-standard path)
Windows 7 Pro 32-bit SP1

Does tingodb support the $and, $or...

Does tingodb support the $and, $or while joining query clauses with a logical AND returns all documents that match the conditions of both clauses.

Support custom `_id`s

Hey there,

I was looking at nedb before, but this looks even better.

Now I was wondering if it would be possible to specify a custom _id when inserting a document. Right now it just throws "Invalid object key (_id)" at me.

This is useful since I'm storing Facebook user information. Their UID is unique enough for me. Sometimes I also store Google Accounts information, in that case, a email is unique enough. (this is a local database without much data)

Basically, using custom ids would simplify some of my schemas and some logic.

Keep it up!

cursor.toArray() fails if only one item in the collection...

Ran in to this in my first attempt to use tingodb...

If you call cursor.toArray() it will fail with "Cursor is closed" because the function isClosed() has this code:

tcursor.prototype.isClosed = function () {
if (!this._items)
return false;
return this._i==this._items.length-1;
}

So if the collection has 1 item this code will fail with the error "Cursor is closed"

    cursor = this.collection.find();
    cursor.toArray(function(error, list) {
        if( error ){
            return def.reject(error);
        }
        else{
            result.list = list;
            def.resolve(result);
        }
    });

_i is 0 and items.length-1 is 0... thus BOOM it's closed and you can't call toArray().

Simultaneous acces

Hello,
I'm considering using tingodb but I was wondering if it allows simultaneous acces ?

For example :
ComputerA et ComputerB are both connected to the network.
Tingodb database file is store in a folder of the network (and so is accessible by both computers)

--> Is it possible that CompA reads database file while CompB is writing in ?

Thanks

Florian

The function find return the cursor instead the data.

What I am testing:

var db = new Engine.Db('db', {});
var movies = db.collection("movies");

movies.find({},function(err,v){
console.log('v: -> ', [err, v]);
})

The output:

v: -> [ null,
{ INIT: 0,
OPEN: 1,
CLOSED: 2,
GET_MORE: 3,
_query: {},
_c:
{ _tdb: [Object],
_name: 'movies',
_store: {},
_fd: null,
_fsize: null,
_id: 1,
_wq: [Object],
_tq: [Object],
_idx: [Object],
_cache: [Object],
_mc: {},
collectionName: 'movies',
_cmaxobj: 1024,
_filename: 'removed the value of this field' }
_i: 0,
_skip: 0,
_limit: null,
_count: null,
_items: null,
_sort: null,
_hint: undefined,
_arFields: {},
_fieldsType: null,
_fieldsExcludeId: false,
_fields: {},
timeout: true } ]

Tks

map/reduce duplicate keys when restart

Consider this program :

var assert = require('assert');

// var Engine = require('mongodb');
// var db = new Engine.Db('test', new Engine.Server('localhost', 27017), {w: 1});
var Engine = require('tingodb')();
var db = new Engine.Db(__dirname, {});

db.open(function(err, db) {
    assert.equal(null, err);

    db.createCollection('collection_1', function(err, collection) {
        assert.equal(null, err);

        collection.insert([
            {'user_id':1, 'type':'A'}
          , {'user_id':2, 'type':'B'}
          , {'user_id':3, 'type':'B'}
          , {'user_id':4, 'type':'C'}
          ], {w: 1}, function(err, r) {

            var map = function () {
              emit(this.type, 1);
            };
           var reduce = function (key, values) {
              var c = 0;
              values.forEach(function (cur) {
                  c += cur;
                }
              );
              return c;
            }

            collection.mapReduce(map, reduce, { out : { replace: 'collection_2' } }, function(err, outCollection) {
                assert.equal(null, err);

                outCollection.find().toArray(function(err, results) {
                    assert.equal(null, err);

                    console.log(results);
                    db.close();
                });
            });
        });
    });
});

Then, run it 2 times.

With mongodb, no problem, but with Tingodb, the second run is broken :

touv@p-touv:~/Tests/tingodb$ node ./mr.js 

assert.js:92
  throw new assert.AssertionError({
        ^
AssertionError: null == {"errmsg":"Error: duplicate key error index"}
    at /home/touv/Tests/tingodb/mr.js:34:24
    at /home/touv/Tests/node_modules/tingodb/node_modules/safe/lib/safe.js:27:11
    at /home/touv/Tests/node_modules/tingodb/lib/wqueue.js:37:7
    at /home/touv/Tests/node_modules/tingodb/node_modules/safe/lib/safe.js:27:11
    at /home/touv/Tests/node_modules/async/lib/async.js:133:21
    at /home/touv/Tests/node_modules/tingodb/lib/wqueue.js:37:7
    at /home/touv/Tests/node_modules/tingodb/lib/tcoll.js:412:12
    at wqueue._exec (/home/touv/Tests/node_modules/tingodb/lib/wqueue.js:36:3)
    at /home/touv/Tests/node_modules/tingodb/lib/wqueue.js:32:9
    at wqueue.first (/home/touv/Tests/node_modules/tingodb/lib/wqueue.js:11:39)

As we can see in the collection's file, tingodb has duplicated lines :

touv@p-touv:~/Tests/tingodb$ touv@p-touv:~/Tests/tingodb$ cat collection_2 
{"k":"0000000080","o":"0000000021","v":"001"}
{"_id":"A","_uid":1,"_dt":1387877760020,"_s":"d7eddd50066bd76ed71cee1bc94d3820"}
{"_id":"A","value":1}
{"k":"0000000080","o":"0000000021","v":"001"}
{"_id":"B","_uid":1,"_dt":1387877760020,"_s":"9fd1daacd77233efd27dfe4398771ba0"}
{"_id":"B","value":2}
{"k":"0000000080","o":"0000000021","v":"001"}
{"_id":"C","_uid":1,"_dt":1387877760021,"_s":"2c4c906c4af00fead9ad3b9e7730e79e"}
{"_id":"C","value":1}
touv@p-touv:~/Tests/tingodb$

Mixed set of projection options (0,1) is valid if only _id is -1 and the rest is true

Hi,

I am getting this error:

Mixed set of projection options (0,1) is not valid

However, according to MongoDb's documentation on _id

The projection parameter takes a document of the following form:

{ field1: <boolean>, field2: <boolean> ... }
The <boolean> value can be any of the following:

1 or true to include the field. The find() method always includes the _id field even if the field is not explicitly stated to return in the projection parameter.
0 or false to exclude the field.
A projection cannot contain both include and exclude specifications, except for the exclusion of the _id field. In projections that explicitly include fields, the _id field is the only field that you can explicitly exclude.

Note the last sentence!
I would send a pull request, but I don't know your codebase at all, and I found the part that throws the error a little intimidating :D

Thanks for listening!

implement "distinct" method

http://docs.mongodb.org/manual/reference/method/db.collection.distinct/

Compacting database

I see that when collection is opened that it is compacted if overuse ratio is exceeded. I am running on a pretty small footprint and would like to know of to configure this setting, plus if it is possible to force compaction of data on demand?

Great concept BTW!

Upsert with $setOnInsert strips data

The following code:

var Db = require('tingodb')({}).Db;

var db = new Db('data', {});
var c = db.collection('test');
c.update(
  {myid: 1},
  {
    myid: 1,
    updated: new Date(),
    $setOnInsert: {
      created: new Date()
    }
  },
  {upsert: true},
  function(er) {
    if (er) { return console.log(er); }
    c.find().toArray(function(er, data) {
      console.log(data);
    });
  });

Has the following output:

[ { myid: 1,
    created: Thu May 15 2014 11:58:50 GMT-0600 (MDT),
    _id: 2 } ]

The expected output is:

[ { myid: 1,
    created: Thu May 15 2014 11:58:50 GMT-0600 (MDT),
    updated: Thu May 15 2014 11:58:50 GMT-0600 (MDT),
    _id: 2 } ]

You'll notice that the updated field is missing from the output. Any other fields that I set in the document to insert are similarly stripped. This happens both on the first run (the insert) as we as subsequent runs (the update).

Escaping regex characters in regex search

Trying to find string /db/get/audit:
Using search: new RegExp('\\/db\\/get\\/audit', 'i')

Testing, with string: /db/get/audit
/\/db\/get\/audit/i.test(/db/get/audit);
Result: true

but sending the exact same data to find it's escaping the escape characters...?

undefined:5
if ((/\\/db\\/get\\/audit/i).test(((obj.when && obj.when.valueOf()) ))) return
           ^
SyntaxError: Unexpected token ILLEGAL
    at tcoll.__find (C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\
lib\tcoll.js:1039:83)
    at C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\lib\tcoll.js:1
082:8
    at wqueue._exec (C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\
lib\wqueue.js:36:3)
    at C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\lib\wqueue.js:
54:10
    at process._tickCallback (node.js:419:13)

I also tried single escape but that just ignored the escapes:
Trying to find string /db/get/audit:
Using search: new RegExp('\/db\/get\/audit', 'i')

Testing, with string: /db/get/audit
//db/get/audit/i.test(/db/get/audit);
Result: true

undefined:6
}
^
SyntaxError: Unexpected token }
    at tcoll.__find (C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\
lib\tcoll.js:1039:83)
    at C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\lib\tcoll.js:1
082:8
    at wqueue._exec (C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\
lib\wqueue.js:36:3)
    at C:\Users\Ganna\git\AdminExtension\app\node_modules\tingodb\lib\wqueue.js:
54:10
    at process._tickCallback (node.js:419:13)

testing code:

var find = [];

//...

searchVal = new RegExp('\/db\/get\/audit', 'i');

var test = '/db/get/audit';
console.log('\nTesting, with string: '+test);
console.log(searchVal + '.test('+test+');');
console.log('Result: ' + searchVal.test(test));

find.push({when:searchVal});

//...

collection.find({'$or': find}, {
    'skip': start,
    'limit': length,
    'sort': sort
}, function(err, cursor){
    //...
});

About the issues I created...

I just wanted to explain quickly why I created those issues.
I wrote a module to create Json Rest Stores which are fully Dojo-compliant. At the beginning, it was all based on Mongo (in fact, Mongoose!). I then started astracting things. I got to the point that I wrote a simple DB layer for my module, simpledblayer. It's meant to do exactly what my JsonRest module needs to do, and do it well.

I wrote a lot of Unit Tests for my module, and they all pass. Then, I ported the Mongo layer to your fantastic TingoDb, which to me is an absolute godsend: it would allow me to basically use/test/whatever JsonRestStores without installing a db server!

JsonRestStores is a piece of the Hotplate puzzle... which is even more exciting.

Anyway, feel free to close this (noisy) ticket, I just wanted to explain why I made those requests. I realise that you might not be actively working on this module right now, or that the fixes I asked for might be complicated.

At this point, I managed to work around the cursor reset (by calling it later in my layer) and the _id in projection issue (by actually deleting _id by hand if it's not part of the schema). However, there is no way for me to get around the regexp issue and the sorting issue. The only pressing one, however, is the regexp one.

Thank you and keep up the good work!

Positional Operator ($) not working

I'm doing something similar to this (context: http://stackoverflow.com/a/12374934/389812):

Model.findAndUpdate(
    { 'pdfs.pdf_id': pdf_id }, 
    { $set: { 
        'pdfs.$.title': title, 
        'pdfs.$.description': description 
    }}, function (err, numAffected) { ... }
);

I'm using Tungus, but I'm guessing it's really a Tingo problem. When I switch to mongodb, it works fine.

If I substitute out the '$' for a '1', does update the first subdocument in the array, but '$' updates the matching subdocument, which is really what is needed.

Sort doesn't work properly (probably only considering one field)

Sort seems to work only partially. I think it only considers the first field.
Have a look at the TingoTicket.js script: sorting only seems to apply to the first field in the sort parameter:

var 
  dummy
, Db = require('tingodb')().Db;
;


var db = new Db('/tmp/tests', {});
var c = db.collection( 'test' );


function printErr( err ){
  if( err ){
    console.log("ERROR!");
    console.log( err );
  }
}

function populate( cb ){

  console.log("Zapping and populating the DB...");
  c.remove( {}, function( err ){
    c.insert( {  name: 'Chiara',    surname: 'Mobily',     age: 22 } , function( err ){
      c.insert( {  name: 'Tony',    surname: 'Mobily',     age: 37 } , function( err ){
        c.insert( {  name: 'Sara',    surname: 'Connor',     age: 37 } , function( err ){
          c.insert( {  name: 'Daniela',    surname: 'Mobily',     age: 37 } , function( err ){
            cb( err );
          })
        })
      })
    })
  })
}

function idProjection( cb ){

  c.find( {}, { name: 1, age: 1, _id: -1 }, function( err, res ){
    if( err ){
      cb( err );
    } else { 
      res.toArray( cb );
    }

  })  
}


function countResetsCursorWorks( cb ){

  cursor = c.find(  { }, { }  );

  cursor.count( function( err, total ){
    if( err ){
      cb( err );
    } else {
      console.log("TOTAL:");
      console.log( total );

      cursor.toArray( function( err, total ){
        if( err ){
          cb( err );
        } else { 
          cursor.count( cb );
        }

      })
    }
  })
}

function countResetsCursorDoesNotWork( cb ){

  //cursor = c.find(  { '$and': [ { name: 'Tony' }, { surname: 'Mobily' }, { age: 37 } ] }, { name: true, surname: true, age: true }   );
  cursor = c.find(  { name: 'Tony', surname: 'Mobily',  age: 37 }, { name: true, surname: true, age: true }   );

  cursor.count( function( err, total ){
    if( err ){
      cb( err );
    } else {
      console.log("RECORDS COUNTED:");
      console.log( total );

      cursor.toArray( function( err, total ){
        if( err ){
          cb( err );
        } else { 
          cursor.count( cb );
        }

      })
    }
  })
}


function sortOnlyOneField( cb ){
  c.find( {}, { sort: { surname: 1, name: 1 }  }, function( err, results ){
    if( err ){
      cb( err );
    } else {
       results.toArray( function( err, a ) {
         console.log("RESULTS SHOULD BE ORDERED BY SURNAME,NAME:");
         console.log( a );
         console.log();
         cb( null );
       });
    }
  });
}

function regexpBroken( cb ){
  // c.find( { '$and': [ { surname: { '$regex': /.*nor$/ } } ] } , function( err, results ){
  c.find( { surname: { '$regex': /.*nor$/ } }, function( err, results ){
    if( err ){
      cb( err );
    } else {
       results.toArray( function( err, a ) {
         console.log("RESULTS SHOULD BE LIMITED:");
         console.log( a );
         console.log();
         cb( null );
       });
    }
  });
}




populate( function( err ){

  idProjection( function( err ){
    printErr( err );

    countResetsCursorWorks( function( err ){
      printErr( err );

      countResetsCursorDoesNotWork( function( err ){
        printErr( err );

        sortOnlyOneField( function( err ){
          printErr( err );

          regexpBroken( function( err ){
             printErr( err );
             db.close();
          })   
        }); 
      });
    });
  });
})

Regexp filters don't work

Regexp filters are meant to work (from what I read in other tickets). And yet, they are failing me.
Have a look at the TingoTicket.js script -- the filter doesn't seem to be applied

var 
  dummy
, Db = require('tingodb')().Db;
;


var db = new Db('/tmp/tests', {});
var c = db.collection( 'test' );


function printErr( err ){
  if( err ){
    console.log("ERROR!");
    console.log( err );
  }
}

function populate( cb ){

  console.log("Zapping and populating the DB...");
  c.remove( {}, function( err ){
    c.insert( {  name: 'Chiara',    surname: 'Mobily',     age: 22 } , function( err ){
      c.insert( {  name: 'Tony',    surname: 'Mobily',     age: 37 } , function( err ){
        c.insert( {  name: 'Sara',    surname: 'Connor',     age: 37 } , function( err ){
          c.insert( {  name: 'Daniela',    surname: 'Mobily',     age: 37 } , function( err ){
            cb( err );
          })
        })
      })
    })
  })
}

function idProjection( cb ){

  c.find( {}, { name: 1, age: 1, _id: -1 }, function( err, res ){
    if( err ){
      cb( err );
    } else { 
      res.toArray( cb );
    }

  })  
}


function countResetsCursorWorks( cb ){

  cursor = c.find(  { }, { }  );

  cursor.count( function( err, total ){
    if( err ){
      cb( err );
    } else {
      console.log("TOTAL:");
      console.log( total );

      cursor.toArray( function( err, total ){
        if( err ){
          cb( err );
        } else { 
          cursor.count( cb );
        }

      })
    }
  })
}

function countResetsCursorDoesNotWork( cb ){

  //cursor = c.find(  { '$and': [ { name: 'Tony' }, { surname: 'Mobily' }, { age: 37 } ] }, { name: true, surname: true, age: true }   );
  cursor = c.find(  { name: 'Tony', surname: 'Mobily',  age: 37 }, { name: true, surname: true, age: true }   );

  cursor.count( function( err, total ){
    if( err ){
      cb( err );
    } else {
      console.log("RECORDS COUNTED:");
      console.log( total );

      cursor.toArray( function( err, total ){
        if( err ){
          cb( err );
        } else { 
          cursor.count( cb );
        }

      })
    }
  })
}


function sortOnlyOneField( cb ){
  c.find( {}, { sort: { surname: 1, name: 1 }  }, function( err, results ){
    if( err ){
      cb( err );
    } else {
       results.toArray( function( err, a ) {
         console.log("RESULTS SHOULD BE ORDERED BY SURNAME,NAME:");
         console.log( a );
         console.log();
         cb( null );
       });
    }
  });
}

function regexpBroken( cb ){
  // c.find( { '$and': [ { surname: { '$regex': /.*nor$/ } } ] } , function( err, results ){
  c.find( { surname: { '$regex': /.*nor$/ } }, function( err, results ){
    if( err ){
      cb( err );
    } else {
       results.toArray( function( err, a ) {
         console.log("RESULTS SHOULD BE LIMITED:");
         console.log( a );
         console.log();
         cb( null );
       });
    }
  });
}




populate( function( err ){

  idProjection( function( err ){
    printErr( err );

    countResetsCursorWorks( function( err ){
      printErr( err );

      countResetsCursorDoesNotWork( function( err ){
        printErr( err );

        sortOnlyOneField( function( err ){
          printErr( err );

          regexpBroken( function( err ){
             printErr( err );
             db.close();
          })   
        }); 
      });
    });
  });
})

sergeyksv / tingodb Goto Github PK

tingodb's People

Contributors

Stargazers

Watchers

Forkers

tingodb's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs