deep populate about camo

[scottwrobinson]

Hi Matt,

Glad you like Camo! It is still in the early stages, so there are definitely some missing features, like support for nested data is partially missing. I say 'partial' because you can save nested data via EmbeddedDocument, but you can't specify a nested schema directly in a Document.

So, as of v0.8.0, you can do this:

class Comment extends EmbeddedDocument {
    constructor() {
        super();
        this.username = String;
        this.body = String;
    }
}

class Post extends Document {
    constructor() {
        super('post');
        this.title = String;
        this.body = String;
        this.comments = [Comment];
    }
}

var post = Post.create({
    title: 'Camo is Fun',
    body: 'This is my post...',
    comments: []
});

var comment1 = Comment.create({
    username: 'scott',
    body: 'What a great post...'
});

var comment2 = Comment.create({
    username: 'bill',
    body: 'This post sucks'
});

post.comments.push(comment1);
post.comments.push(comment2);

post.save().then(function() {
    // ...
});

But you can't do this (yet):

class Post extends Document {
    constructor() {
        super('post');
        this.title = String;
        this.body = String;
        this.comments = [{
            username: String,
            body: String
        }];
    }
}

The main difference is where the Comment schema is declared.

Hopefully the EmbeddedDocument will work out for you for now. If you have any more questions it might help if I saw the schema you're trying to create and I'll give you hints on how to do it. Otherwise I'm working on better nested data support so hopefully your problems will be solved soon.

Scott

[MattMcFarland]

Hey, thanks for the reply!

Yeah I was thinking about using Embedded Document but really the Comment Class is extended off of a Document class. Reason is there are other document classes that have similar functionality, so in order for me to use embedded document, I would have to duplicate a lot of code. Really trying to avoid that right now, in fact spent 6+ hours trying to haha.

Here are my classes

Base: extends Document  (contains methods like voting, flagging, editing, removing, undo removing etc)

      Actionable:  extends Base (but actionable has the addComment method added)

      Comment: extends Base 

      TutorialRequest: extends Actionable (Schema has a comment: [Comment])

      TutorialSolution:  extends Actionable (Schema also has a comment: [Comment])

So it really is not a good use case for my scenario to use EmbeddedDocument for Comments, as they too can be voted, flagged, edited, removed, etc and I want to stay DRY. So they are really Documents. They are added to the schema by being typed as Comment.

I did look at the Embedded Doc class and the code used to create the embed document (looks like it replaces the ref with actual doc maybe, permanently??) is quite complicated so then I didnt want to copy that either.

I ended up trying to use Promises to populate the code, but no matter what I do, it seems like it either creates a ton of copies of comments, or creates a weird nested array. It just wasn't working! (I'm new to es6 promises, so there's that..)

I do thank you for your time and I am a huge fan of Camo. IT is really been amazing to use, much better than Mongoose (syntactically)

I'll try to look at my code again, but really want to avoid Embed if I can. I'll also share if I can make any headway. So far it's been very challenging (which is good too :)) - You can actually see my adventures with camo (and struggles) https://www.livecoding.tv/mmcfarland/

[MattMcFarland]

So anyway, Here's the code I've struggling with:

 get DTO () {
    //at one point i had a console.log and an i++, j++ iterator to see how many times things were actually being added
    return new Promise((resolve, reject) => {
      var solutionPromises = this._values.solutions.map((sol) => {
        // I've also tried the var solution = Object.assign here.
        return Promise.all(sol._values.comments.map((com) => {
          var solution = Object.assign(sol.DTO, {
            comments: []
          });
          return new Promise((res, rej) => {
            Comment.loadOne({_id: com}).then((comment) => {
              // solutions with comments end up looking like this solutions: [ [ object ] ] (why?)
              // was using comments[index] to assign the comment to, also trie solutions.push
              // and get the same results.. is the problem Object.assign????
              solution.comments[solution.comments.length] = (Object.assign(comment, comment.DTO));
              // also tried res(sol), which actually was an improvement but I ended up rage git reset --hard
              res(solution)
              // main problem is the comments are duplicating and are not being assign to the right
             // solutions.  it seems the last comment is being applied multiple times.  which I guess may
             // be the object is getting mutated.
            })
          })
        }))
      });

      Promise.all(solutionPromises).then((solutions) => {
        resolve({
          solutions,  // using lodash _.flatten here fixes the [[ object ]] issue but would prefer
                           // prevent that from happening in the first place.
          id: this.id,
          title: this.title,
          linkMeta: this.linkMeta,
          authorName: this.authorName,
          authorUrl: this.authorUrl,
          editorName: this.editorName,
          editorUrl: this.editorUrl,
          flags: this.flags,
          score: this.tallyVotes(),
          comments: this._values.comments
        }, this._values)
      });
    });

  }

That is basically my "Question" class - as I'm creating a Q&A website. (but my questions are known as tutorial requests, and answers are known as tutorial solutions)

Since Questions and Answers can both have comments, it's really the question replies that get screwed (as they are a child of questions)

So the code above is really my 20th attempt (or so) at getting the comments to appear. Like I said, I do get them to appear, but i end up making lots of copies. Like the example above somehow makes a nested array of "solutions" then the comments for the solutions somehow get copied - what I mean is there's only two comments and one is on one solution each, but somehow I end up with like 8 comments and they're all the 4th comment (maybe I am doing some pass-by-value mutation)

[scottwrobinson]

Hmm I'm not sure I'm following your code exactly, but I think I get what you're trying to do. Are you just trying to load all comments in each Solution?

A couple things:

• A Proxy is used in the background of Document to access your data from _values. It's the thing that makes this.data === this._values.data work.
• I would advise against accessing data directly via this._values. If you want to get the values stored in sol, just use this.sol and the Proxy object will handle retrieving it for you. The Proxy doesn't do a whole lot right now, but it will in future versions.
• I would also advise against merging objects via Object.assign (although I'm not really experienced on how it works). I say this mostly because I don't know how it affects the Document's use of Proxy during assignment/retrieval.
• As of v0.8.0, all 1-deep embedded and referenced documents are already populated for you. So if you have something like Solution.comments then using loadOne will already populate the comments for you, but Solution.comments[0].comment would not be populated. You'll have to load 2-deep (and greater) documents yourself.

Also, the part where you're doing...

var solution = Object.assign(sol.DTO, {
    comments: []
});

...I think you're reassigning comments to an empty array in sol for every comment you iterate over.

Do your DTO methods just populate referenced data? If so, why are you doing Object.assign(comment, comment.DTO) after already loading the comment using loadOne? To get the further nested comments?

I might be of more help if I could see the actual schema for TutorialSolution and Comment.

Some of the problems of loading nested data like this is because of Camo's deficiencies, so this will be a good use case to keep in mind during development. Thanks!

[MattMcFarland]

Hey Scott - I really appreciate your time analyzing my code. You've made some good points regarding Object.assign, and I haven't really messed with Proxies so I can say that I am not sure either. I can say that Object.assign wasn't behaving as expected so I've removed it.. However the var solution = [] creates a new "throw-away" instance of an empty array that I was attempting to use as an array to write to, so I dont see how that would create a nested array unless I am missing something?

One thing you said here:

You'll have to load 2-deep (and greater) documents yourself

That is exactly what I'm trying to do, and have been failing at it. The 1-deep comments load just fine. The main issue is that loadOne is asynchronous so when you try to use loadOne to grab a nested doc .then() will execute well after a return, meaning the return itself needs to be a Promise. The code I shared previously was an attempt at just that, but a failure of one. I think I'll retry again here tonight...

So last night I decided to refactor the DTO getters, - I cleaned them up with the goal in mind to make it easier to retry wrapping loadOne in a promise so that 2-deep objects are returned. Either that, or maybe it is possible to modify loadOne to allow for deeply populated items, but I think it might be a good use-case for a camo plugin and not necessarily a part of camo core itself.

Base Class

Here's the Base class, it is extended by other documents and contains common schema, getters, and methods.

"use strict";
// Base Model
const
  _        = require('lodash'),
  Document = require('camo').Document,
  Utils    = require('../utils');

class Base extends Document {
  constructor(name) {
    super(name);

    this.schema({
      author:   { type: Object },
      editor:   { type: Object },
      title:    { type: String },
      content:  { type: String },

      /* Flagging */
      flaggers:         { type: [Object]},

      /* Voting */
      voters: { type: [Object] },

      /* Meta */
      created_at: { type: Date, default: Date.now },
      updated_at: { type: Date, default: Date.now },
      removed:    { type: Boolean, default: false}
    });

  }



  get flags () {
    return {
      spam: this.countFlags('spam'),
      offensive: this.countFlags('offensive'),
      vague: this.countFlags('vague'),
      duplicate: this.countFlags('duplicate')
    }
  }
  get authorName () {
    if (!this.author) {
      return '';
    }
    return this.author.fullName;
  }

  get editorName () {
    if (!this.editor) {
      return '';
    }
    return this.editor.fullName;
  }

  get authorUrl () {
    if (!this.author) {
      return '';
    }
    return 'users/' + Utils.Users.getId(this.author);
  }

  get editorUrl () {
    if (!this.editor) {
      return '';
    }
    return 'users/' + Utils.Users.getId(this.author);
  }

  countFlags (flagType) {
    return _.countBy(this.flaggers, {type: flagType}).true || 0;
  }

  addOrRemoveFlag (user, flagType) {
  // ommitted for brevity (works fine) 
  }

  createOrUpdateVote (user, direction) {
     // ommitted for brevity (works fine) 
  }

  tallyVotes () {
     // ommitted for brevity (works fine) 
  }

  removeOrUndoRemove (editor) {
    // ommitted for brevity (works fine) 
  }

}

module.exports = Base;

Actionable Class

Actionable class extends Base class. TutorialRequest and TutorialSolution extend Actionable, Comment extends Base. This is so that all extended classes have access to my common getter methods, common api related methods, and common schema.

Originally I tried to have just a Base class, but since comments extend Base I had to create Actionable.

Side Note:

You might find its worthy noting here that constuctor(name), super(name) needs to be in both Base and Actionable, otherwise _meta.collectionName will be incorrect. If you have a Document that extends another Document which extends another Document, you should "propagate" the name up the "extend chain", or else _meta.collectionName will equal the top level document instead of the target.

"use strict";
// Base Model
const
  _        = require('lodash'),
  Base     = require('./Base'),
  Comment  = require('./Comment'),
  Utils    = require('../utils'),
  ObjectID = require('mongodb').ObjectID;

class Actionable extends Base {
  constructor(name) {
    super(name);

    this.comments = {
      type: [Comment]
    };

  }
  addComment (author, message) {
    return new Promise((resolve, reject) => {
      Comment.create({
        author,
        message: Utils.xss(message)
      }).save().then((comment) => {
        this.comments.push(comment);
        this.save()
          .then(() => resolve(comment))
          .catch((e) => reject(e))
      }).catch((e) => reject(e))
    });
  }
}

module.exports = Actionable;

Comment DTO / Schema

New and improved

  constructor() {
    super('comments');
    this.message = String;
    this.comments = [Comment]; // doing a comment on a comment lol
  }
  get DTO () {
    return {
      type: 'Comment',
      id: this.id,
      authorName: this.authorName,
      authorUrl: this.authorUrl,
      editorName: this.editorName,
      editorUrl: this.editorUrl,
      message: this.message,
      flags: this.flags,
      score: this.tallyVotes()
    }
  }

TutorialRequest DTO / Schema

New and improved
Note: the map works fine, of course though when sol.DTO runs it wont get the comments as they are not populated, instead they are document references (which is understandable)

  constructor() {
    super('tutorialrequests');
    this.engine = String;
    this.version = String;
    this.solutions = [Solution];
    this.tags = [Tag]; // tags to be implemented later.
    this.permalink = String;
  }

 get DTO () {

    return {
      type: "TutorialRequest",
      solutions: this.solutions.map((sol) => {
        return sol.DTO;
      }),
      id: this.id,
      title: this.title,
      linkMeta: this.linkMeta,
      authorName: this.authorName,
      authorUrl: this.authorUrl,
      editorName: this.editorName,
      editorUrl: this.editorUrl,
      flags: this.flags,
      score: this.tallyVotes(),
      comments: this.comments.map((com) => {
        return com.DTO;
      })
    };


  }

TutorialSolution DTO / Schema

New and improved

  constructor() {
    super('tutorialsolutions');
    this.linkMeta = {type: Object};
  }
  get DTO () {
    return {
      type: 'TutorialSolution',
      id: this.id,
      authorName: this.authorName,
      authorUrl: this.authorUrl,
      editorName: this.editorName,
      editorUrl: this.editorUrl,
      flags: this.flags,
      score: this.tallyVotes(),
      comments: this.comments.map((com) => {
        if (com.DTO) {
          return com.DTO;
        } else {
          return Comment.loadOne({_id: com})
            .then((m) => m.DTO)
// actually m.DTO (the comment) is populated at this point, but the object is never returned
 // and instead an empty object is sent back, this is because by the time we populate
// with loadOne, the item si already sent back.  The only way to get around this is by adding 
// a promise, but I tried this, I spent hours trying to add promises and it just got really messed up
            .catch((e) => Utils.Log.error(e));
        }
      })
    }
  }

So basically when I download a tutorial request, I get the array of solutions which also have arrays of comments themselves (2 levels deep) - this is where mongoose plugin populateDeep would come in handy.

The issue is that loadOne returns a promise, but my DTO just returns the objects so the api returns the objects before the loadOne promise is completed. So I've figured that all I need to do is wait for loadOne and then return the objects to the api - and I believe that it is possible and maybe the root of my problem was using Object.assign, etc. I am going to try again.

One cool thing I might try is extending loadOne with a special object like loadOneAndDeepPopulate or perhaps loadOne(xyz, deepPopulate: true) or something so that the initial load will already have the 2-deep items populated.

Thanks again, hope this makes sense.

If you want, we can set up a stream session and I can show you what I am talking about. I'd love to help make camo better and also learn how to use it more effectively and livecoding is fun. I've also made you a collaborator to my private project on github that uses camo. just in case...

[scottwrobinson]

Hi Matt,

That's one of the fundamental problems with JavaScript being asynchronous, if a method makes an async call, then it has to return a promise, no way around it.

It would definitely be nice if Camo could handle deep population. I'm still working on a way to let you tell Camo which fields to populate (and how deep) and which to leave as references. Hopefully I'll have the feature out soon. I'll take a look at Mongoose's populateDeep plugin. Thanks for pointing that out!

I should have more time to look at your code later this week. I'll get back to you then. Also, thanks for your feedback on Camo! It really helps to have another set of eyes and to see how people are using it.

Scott

[MattMcFarland]

Hey man,

I finally got this working! The solution below is a bit ugly, so I plan on cleaning it up later.. For now, I'm leaving it as is seeing as how I spent over 10 hrs on this problem..

exports.getById = (M, req, res, Comment) => {
  return M.loadOne({_id: ObjectID(req.params.id)})
    .then((m) => {
      var
        done = false,
        solutions = _.clone(m.DTO.solutions),
        commentsLoaded = false,
        loop = (solIndex) => {
          var
            index = solIndex || 0,
            sol = solutions[index],
            j = 0;

          if (sol && sol.comments && sol.comments.length) {
            if (!sol.processing) {
              j = 0;
              sol.processing = true;
              //console.log('solution [' + index + '] has', sol.comments.length, 'comment(s)');
              sol.xcomments = Promise.all(sol.comments.map((c, i) => {
                //console.log('get comment', c);
                return Comment.loadOne({_id: c});
              })).then((g) => {
                commentsLoaded = true;
                sol.comments = g.map((com) => {
                  return com.DTO;
                });

                //console.log(g.length, 'comments loaded');
                if (done) {
                  res.json(Object.assign(m.DTO, {solutions}));
                }
              })
            } else if (done) {
              //console.log('but we done');
            } else {
              //console.log('waiting...');
            }
          } else {
            commentsLoaded = true;
            //console.log('solution [' + index + '] has no comment(s)');
          }
          if (!commentsLoaded) {
            _.defer(loop, 500);
          } else {
            if (index >= solutions.length - 1) {
              done = true;
            } else {
              loop(index + 1);
            }
          }
        };
      if (solutions.length) {
        //Utils.Log.info('found ' + solutions.length + ' solutions');
        loop();
      } else {
        res.json(m.DTO);
      }
    })
    .catch((e) => Utils.Log.error(e));
};

[scottwrobinson]

Awesome! Glad you got it working. Hopefully I can add extend Camo so you won't have to go through all of this again. Thanks again for all the feedback and everything.

I'm going to close this issue for now, but let me know if you need anything else.

Scott