A bootstrap library to import javascript objects into mongodb using a series of raw json files
I started my career as a full-stack engineer doing LAMP development in NYC and worked at several successful startups. I transitioned to engineering leadership and eventually went onto co-found gitBabel a knowledge & learning platform that enables document-driven cultures.
I also provide mentoring and consulting services, feel free to connect with me at the following:
It seems that Mongoose-fixture is trying to access all the files in the schema folder. In my opinion, it isn't necessary to do that, it should look up only the schema defined in mongoose-fixture-config.js:
var allFixtures = [
{
schema:'Product',
data:'product',
collection:'products'
}
];So, in this case, it should look up only the product.js placed in schema folder. When it looked up all the schema files, it ran to history.js before product.js. This is the cause of the problem.
// history.js
module.exports = function(mongoose, conn) {
var ProductSchema = mongoose.model('Product').schema;
var HistorySchema = new mongoose.Schema({
...
products: {
type: [ProductSchema]
}
});
mongoose.model('History', HistorySchema);
return {
name: 'History',
schema: HistorySchema
};
};This is my point of view, it may be wrong. Thanks in advance for your suggestion on how to solve it.
Ran into an issue where a set of models need to have a set of models set as a collection of defaults. However, because of the circular reference and order of operations it isn't feasible to set the defaults during initial creation. Need a way to have a fixture that isn't about inserting new records, but purely about updating and creating references to other fixtures. The boilerplating from generateFixture will not work because it sends an array of object literals to the callback to be process. So the return value should be of a different data shape and the processor needs to be able to reflect the different data and handle the update operation.
There are several possible implementation options
insert or updatein basefixtureloader
var fixtureModel = mongoose.model(fixtureItem, fixtureSchema);
so looks like model name is the fixture.itemName.
in mongoosefixture
if(name === item.collection){
collectionExists = true;
}
for deletes
adding seems to use fixture.itemName and remove seems to use the fixture.collection. If I set these two the same, everything works fine. If they are different, the remove code breaks.
Currently MongooseFixtureCommands is built on top of substacks command library. I should move over to TJ's commander library. It provides a better api abstraction with improved help/documentation from cli
The approach shown in #5 (comment) is no doubt very versatile, but it seems to rely on data that already exists in the database and requires some existing knowledge of how the library works in order to use it.
The option to use an interface that supports the following construct that would make the learning curve very shallow and, consequently, allow far greater adoption.
// fixture object - single letter for brevity
var o = {};
// collections
o.families = {};
o.people = {};
// documents
o.families.smith = { name: "Smith" };
o.families.jones = { name: "Jones" };
o.families.connors = { name: "Connors" };
o.people.peter = { name: "Peter", family: o.families.smith };
o.people.catherine = { name: "Catherine", family: o.families.connors, parents: [ { name: o.people.peter.name } ] };
console.log("Peter's family: " + o.people.peter.family.name);
console.log("Catherine's family: " + o.people.catherine.family.name);
console.log("Catherine's parent(s): " + o.people.catherine.parents[0].name);Note how the collections are declared, which could be determined using lodash's _.keys(object) method).
Obviously, a full example showing self-references and references between collections where the data is purely described in text files or scripts would also be helpful after the implementation.
There maybe some benefit to separating the logic code and fixture data objects into separate files, to make it simpler to understand, possibly with them being loaded using async.series (already provided in a boilerplate).
This interface may well be quite difficult to implement or there may be some reasons you can think of why it wouldn't be practical or versatile which outweigh it's ease of use. If so, and if you do not object, I think it's worth discussing.
Tests just should harness into grunt file
Hi,
during develop my project I regnocnized a difference between the expected schemas for mongoose [1] and mongoose fixture [2]:
[1]
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
module.exports = mongoose.model('User', new Schema({
name: String
}));
[2]
module.exports = function (mongoose) {
return {
name: 'User',
schema: mongoose.Schema({
name: String
})
};
};
My problem is that I can't find a schema that fits for both.
What did I wrong?
Thanks for helping me :)
Cheers Tobias
Hook up jshint to prevent further low level js syntax errors
Use case would be a server being able to load/reset fixtures for a given event.
In the future we want to build a migration [schema/data] system.
Current ideology is south for django
I installed this using
sudo npm install -g mongoose-fixture
Then I ran
mongoose-fixture --generateConfig
And I get a stack trace with an error saying:
path.js:313
throw new TypeError('Arguments to path.resolve must be strings');TypeError: Arguments to path.resolve must be strings
at Object.exports.resolve (path.js:313:15)
at Object.realpathSync (fs.js:1207:18)
at run (/usr/local/lib/node_modules/mongoose-fixture/lib/MongooseFixtureCommand.js:78:34)
at Object. (/usr/local/lib/node_modules/mongoose-fixture/bin/mongoose-fixture:7:41)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)
at Function.Module.runMain (module.js:497:10)
at startup (node.js:119:16)
I get the same error when running other commands as well. Version is 0.1.3.
When using the --reset or --remove flags from CLI a missing collection in mongodb will cause the process to terminate and throw an error. This was the result of previous adjustments to improve error handling/messaging.
The problem with the zealous exception handling is it blocks the --reset from continuing if a single removal fails, leaving the mongodb data in an undetermined state at the point of exception. This isn't ideal at all especially in cases where several fixtures can share the same collection resulting it dumping the collection and on subsequent processing the collection no longer exists resulting in an exception. Furthermore, a misnamed collection name could hold up an othewise functioning fixture set during reset.
Fix currently is to warn on stdout which collection failed to remove and to propagate the execution stack.
Found an issue with creating multiple fixtures using the same model. BaseLoader would have an exception raised by mongoose.
Have an issue with creating fixture-listings that then async load other fixtures that are outside of the fixture-listing because the mongoose connection obj is missing the models. Best course is to load all the models on init, solves various issues with very little performance overhead.
The boilerplate schema file generates a file that contains a comment such as the following:
/*
* File: PlanetStructures.js
* Generated by: Mongoose-Fixture (v0.1.3)
*
*/While the auto-generated config and schema files are self-explanatory from the file names they are created with, there is no way to tell from the contents of the fixtures files, meaning that when one is pasted and shared online, it is not immediately clear which of the 3 types of files it is.
For example:
#5 (comment)
Ideally, the comments for all 3 boilerplates would be amended to be consistent and thus all would include the purpose of the file in the comment, but the fixtures boilerplate is the most important.
I would have submitted a patch for this, but as it's such a trivial change and you may have a preference about how the comment is expressed, I will leave it at this issue/ticket. I hope that's ok! :)
Thanks a lot!
First of all, thanks for the great module. I am curious that when I run the mongoose-fixture --fixture='all' --add, is there a way I can add image files into by DB using GridFS?
To load fixture data from one model into another fixture model the method is to use raw asyn.series and grab the necessary models, then create an array of execution that all flows to a callback that has named references. The code is then constantly duplicated and meticulous maintained.
Create an interface helper to solve this problem so that code can be re-used and that implementors can extend the interface for even more modularity.
Current fixture example using async to load clients and templates before creating a fixture for creatives.
var clientModel = conn.model('Client');
var arpClient = clientModel.find({name:'ArpaNet'});
var templateModel = conn.model('Template');
var baseTemplate = templateModel.find({name:/Base/});
var advTemplate = templateModel.find({name:/Advance/});
async.series([
// load client
function(callback){
arpClient.exec(function(err, client){
return callback(null, client);
});
},
// load a template
function(callback){
baseTemplate.exec(function(err, templates){
return callback(null, templates);
});
},
function(callback){
advTemplate.exec(function(err, templates){
return callback(null, templates);
});
}
//////////
// End series, invoke callback
//////////
], function(err, results){
/*
* results[0] -> client
* results[1] -> baseTemplate
* results[2] -> advTemplate
*/
var arpClient = results[0],
baseTemplates = results[1],
advTemplates = results[2];
fixture.push([
{
client : arpClient.id,
template:{
name:baseTemplates.name,
_id:baseTemplates._id
}
},
{
client : arpClient.id,
template:{
name:advTemplates.name,
_id:advTemplates._id
}
}
]);
return rootCallback(err, fixtures);
}; // end async closureSimplified version
fixtureHelperService.load({
// a collection of mongoose queries to run before doing fixture callback
'populate':[
{id:'arpClient', model:'Client', where:{name:'arpClient'}},
{id:'baseTpl', model:'Template', where:{name:/baseTemplate/}},
{id:'advTpl', model:'Template', where:{name:/advTemplate/}},
], function(err, results){
fixture.push({
client : result['arpClient'].id,
template:{
name:result['baseTpl'].name,
_id:result['advTpl'].id
}
});
return rootCallback(err, fixture);
}
});
Need a way to connect to a mongo cluster. The cluster has one writer and (in our case) 2 readers, so I can get seeding working by adjusting the IP of writer each time I load fixtures, but this is far from idea.
The url I'm using is of the form
mongodb://ip1:port1,ip2:port2,ip3:port3/db_name
The line in question seems to be:
https://github.com/mgan59/mongoose-fixture/blob/master/lib/MongooseFixtureCommand.js#L272
mongoose-fixture --fixture='all' --add
Loading fixtures from - mongoose-fixture-config.js
Established Connection MongoD (localhost:27017)
Mongoose-Fixture init schemas/models into conn
/usr/local/lib/node_modules/mongoose-fixture/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:286
throw err;
^
TypeError: Cannot call method 'create' of undefined
at async.forEach.msg (/usr/local/lib/node_modules/mongoose-fixture/lib/BaseFixtureLoader.js:90:34)
at /usr/local/lib/node_modules/mongoose-fixture/node_modules/async/lib/async.js:99:13
at Array.forEach (native)
at _each (/usr/local/lib/node_modules/mongoose-fixture/node_modules/async/lib/async.js:32:24)
at Object.async.each (/usr/local/lib/node_modules/mongoose-fixture/node_modules/async/lib/async.js:98:9)
at /usr/local/lib/node_modules/mongoose-fixture/lib/BaseFixtureLoader.js:86:19
at module.exports (/var/www/dev/mysite/fixtures/Users.js:26:12)
at module.exports (/usr/local/lib/node_modules/mongoose-fixture/lib/BaseFixtureLoader.js:36:76)
at async.eachSeries.conn.close.disconnectMessage (/usr/local/lib/node_modules/mongoose-fixture/lib/MongooseFixture.js:67:17)
at iterate (/usr/local/lib/node_modules/mongoose-fixture/node_modules/async/lib/async.js:123:13)
cat fixtures/Users.js
var sha1 = require('sha1');
module.exports = function(mongoose, conn, callback){
// standard callback error
var error = null;
// create your data documents using object-literals
var fixture = [];
/*
* Example of adding a data document/fixture item
*/
fixture.push({
// by not defining an _id mongoose-fixture
// will by default set a mongo ObjectID
// defining one manually will override mongoDB
username: 'admin',
password: sha1('admin'),
email: '[email protected]',
role: 'admin'
});
// mongoose-fixture expects implementor to return
// the callback passed in context
return callback(error, fixture);
};
cat schemas/UserSchema.js
module.exports = function(mongoose, conn) {
var UserSchema = new mongoose.Schema({
_id : { type: String },
username : { type: String, required: true },
password : { type: String, required: true },
email : { type: String, required: true, index: { unique: true } },
role : { type: String, default: "normal" },
token : { type: String, default: 0 },
expired : { type: Date, default: Date.now },
created : { type: Date, default: Date.now }
});
return {
name: 'User',
schema: UserSchema
};
}
cat mongoose-fixture-config.js
// Load the default object that helps manage a FixtureConfig
var FixtureConfig = require('mongoose-fixture').FixtureConfig;
var config = require('config');
// Create our fixture config with defined
// mongo-connection and file paths
var fixtureConfig = FixtureConfig({
mongoConnection:{
'host': config.host,
'port': config.port,
'dbname': config.dbname
},
paths:{
schemaPath:__dirname+'/schemas/',
dataFixturePath:__dirname+'/fixtures/'
}
});
// Create a Listing of fixtures
var allFixtures = [
{
itemName: 'User',
schema: 'UserSchema',
data:'Users',
collection:'users'
},
];
// load fixture listings
fixtureConfig.fixtureListings.set('all', allFixtures);
// export the config
module.exports = fixtureConfig;
Thanks for the hard work you've put into this project.
As there is only one example, and it doesn't demonstrate how fixtures containing references can be loaded, here's what I've been using. Feel free to add, or improve it and add it, to your repo.
// Load the default object that helps manage a FixtureConfig
var FixtureConfig = require('mongoose-fixture').FixtureConfig;
// Create our fixture config with defined
// mongo-connection and file paths
var fixtureConfig = FixtureConfig({
mongoConnection:{
'host':'localhost',
'port':'27017',
'dbname':'familydb'
},
paths:{
schemaPath:__dirname+'/model/schemas/',
dataFixturePath:__dirname+'/model/fixtures/'
}
});
// Create a Listing of fixtures
var allFixtures = [
{
// general name used in output log
itemName:'Family',
// name of the schema file (without the .js)
schema:'FamilySchema',
// name of the data-fixture file (without the .js)
data:'Family',
// collection name in for removal process
// Note that mongoose will pluralize this name to "families" when it creates the collection
collection:'family'
}
];
// load fixture listings
fixtureConfig.fixtureListings.set('all', allFixtures);
// export the config
module.exports = fixtureConfig;var ObjectId = require('mongoose').Schema.Types.ObjectId;
module.exports = function(mongoose){
var exportSchema = mongoose.Schema({
// the _id property, and ObjectId variable definition above, can be omitted, as it will be created automatically
_id:{type:ObjectId},
name:{type:String},
parents:{type:Array}
});
return exportSchema;
};// @callback must be returned
// expects (err, object)
// object can be an array of data-documents, or a kwarg['dataFixtures']
var _ = require('underscore');
var ObjectId = require('mongodb').BSONNative.ObjectID;
// Holds fixture data, before assigned to fixture variable in module.exports function
var o = {};
// Creating an object to store the collection rather than pushing immediately to
// an array makes the `parents[]._id` references more meaningful and easier to debug
o.michael = { _id: new ObjectId(), name: "Michael" };
o.fred = { _id: new ObjectId(), name: "Fred" };
o.mildred = { _id: new ObjectId(), name: "Mildred" };
o.andrew = { _id: new ObjectId(), name: "Andrew" };
o.george = { _id: new ObjectId(), name: "George", parents: [ { _id: o.michael._id } ] };
o.sally = { _id: new ObjectId(), name: "Sally", parents: [ { _id: o.fred._id }, { _id: o.mildred._id } ] };
o.peter = { _id: new ObjectId(), name: "Peter", parents: [ { _id: o.andrew._id }, { _id: o.mildred._id } ] };
// Add each property of `o` to an array
var fixtureArray = _.values(o);
module.exports = function(mongoose, callback){
// standard callback error
var error = null;
// create your data documents using object-literals
// var fixture = [];
var fixture = fixtureArray; // see above
// mongoose-fixture expects implementor to return
// the callback passed in context
return callback(error, fixture);
};blank-slate for just config and not fixtures/schemas
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.