meteor-community-packages / denormalize Goto Github PK

Hi,

My case:

Documents from the Users collection has been denormalized into an authorCache field in various different other collections. If a User creates a Question, then that Question-document will have an authorCache field with the Users emails.

When I update the Users emails, all their Question-docs get updated - as expected. However, if there are thousands of docs, then this takes time.

So, I was hoping that Users.update(selector, modifier, {upsert: true}, callback ), when ran on the server, would make the update async and non-blocking. This is the expected behaviour according to Meteor docs.

But, this is not the case when using herteby:denormalize (via cult-of-coders:grapher).

Is there a way to make the Users.update() call return immediately, even when caches must be updated?

Some recommendations (something I will also do this in grapher)
export {Types} from this package and let's use
Types.ONE
Types.ONE_INVERSE
etc, instead of relying on strings.

Hello Herteby,

thank you for the great plugin.
The only thing I missed is client side denormalisation.
Is there an implementation?

Kind regards
Lukas

Hi!

I have before.find and before.update hooks in place to realize authorization on my collections.

Normally I use .direct to bypass those hooks in the migrations. Is there a possibility to use .direct when running the migrate() function?

Thanks!

If one collection is referencing nested fields as such :

Task.cache({
    type: 'one',
    collection: Meteor.users,
    fields: ['profile.firstName', 'profile.lastName'],
    referenceField: 'userId',
    cacheField: 'userCache'
})

Then an update of the profile will trigger unwanted cascading updates that could result in poor application performance depending on collection sizes :

Meteor.users.update({_id: "some id"}, { '$set': { 'profile.connexionDate': new Date() } })

I will try to work on a PR to fix this. Any hints on a proper way to treat this issue will be appreciated

Is your feature request related to a problem? Please describe.
Support Meteor 3.0.

Describe the solution you'd like
Switch to *Async methods.

Describe alternatives you've considered
n/a

Additional context
I think we'd like to merge as much as possible from the open PRs and then start working on Meteor 3.0 support.

That is, only if you want to. :)

@theodorDiaconu
I just realized I had forgotten this type! Since you can use "inversedBy" on both "one" and "many".

I recommend 4 spaces instead of 2 for indentation because it forces you to avoid spaghetti.

It would be an interesting addition to be able to provide a selector for the parent collection. For example:

// Only add a chairCount on furniture of type "table"
Furniture.cacheCount({
  collection: Furniture,
  referenceField: 'furnitureLink',
  cacheField: 'chairCount',
  parentSelector: { type: 'table' },
  selector: { type: 'chair' },
})

// Only add a legCount on furniture of type "chair"
Furniture.cacheCount({
  collection: Legs,
  referenceField: 'furnitureLink',
  cacheField: 'legCount',
  parentSelector: { type: 'chair' },
})

This allows "polymorphic" collections to have different caches depending on their category, type, etc.

It is mostly done for keeping things clean, but one could also imagine using different cache configurations with the same name:

Here's a sarcastic example 🤣:

Users.cacheCount({
  collection: Users,
  referenceField: 'friendIds',
  cacheField: 'americanFriends',
  parentSelector: { citizenShip: 'american' },
  selector: { citizenShip: 'american' },
})

Users.cacheCount({
  collection: Users,
  referenceField: 'friendIds',
  cacheField: 'americanFriends',
  parentSelector: { citizenShip: 'european' },
  selector: { citizenShip: { $in: ['north-american', 'south-american', 'american'] } },
})

You would be responsible for maintaining non-overlapping parent selectors of course, otherwise it will not work.

@Floriferous @copleykj @doronrk

Hi! Sorry for taking so long to respond. I switched jobs about 2 years ago and haven't worked with Meteor since. I was wondering if any of you who are using the package would like to take over the ownership of it?

Maybe I should involve @theodorDiaconu as well, as Grapher is using this package.

I'm trying to get a cache working where the relationship is "many-to-one unique", but I think this package doesn't handle it. Here's an example:

I have a collection of Authors, and a collection of Comments. The authors store an array of commentIds, and I want each comment to denormalize the author, without it being an array.

Authors // { commentIds: ['commentId1', 'commentId2'] }

// What I want
Comments: // { authorCache: { _id: 'authorId', name: 'John' } }

// What I get
Comments: // { authorCache: [{ _id: 'authorId', name: 'John' }] }

Usually many-to-one means there can be many authors linking to the same comment, but that's not the case here, so the cache is not correct.

This package uses import _ from 'lodash', instead it should import only the functions it needs.

This currently constrains the collection hooks package not to upgrade past 0.8.4 and the current version is 1.0.1.

I'd be happy to submit a pull request if there is a chance it will get merged and published.

I'm using this package through grapher, and when I define a cache with bypassSchema: true, this package throws the following error:

TypeError: Cannot read property 'insert' of undefined
W20190315-09:56:31.177(1)? (STDERR)     at ns.Collection.Mongo.Collection.cache (packages/herteby:denormalize/cache.js:112:28)
W20190315-09:56:31.177(1)? (STDERR)     at Linker._initDenormalization (packages/cultofcoders:grapher/lib/links/linker.js:444:33)
W20190315-09:56:31.177(1)? (STDERR)     at new Linker (packages/cultofcoders:grapher/lib/links/linker.js:28:14)
W20190315-09:56:31.177(1)? (STDERR)     at _.each (packages/cultofcoders:grapher/lib/links/extension.js:23:28)
W20190315-09:56:31.178(1)? (STDERR)     at Function._.each._.forEach (packages/underscore.js:147:22)
W20190315-09:56:31.178(1)? (STDERR)     at ns.Collection.addLinks (packages/cultofcoders:grapher/lib/links/extension.js:14:11)
W20190315-09:56:31.178(1)? (STDERR)     at links.js (imports/core/api/loans/links.js:14:1)
W20190315-09:56:31.178(1)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:336:7)
W20190315-09:56:31.178(1)? (STDERR)     at Module.require (packages/modules-runtime.js:238:14)
W20190315-09:56:31.178(1)? (STDERR)     at Module.moduleLink [as link] (/Users/Florian/.meteor/packages/modules/.0.13.0.1aurolz.v3oz++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/reify/lib/runtime/index.js:38:38)
W20190315-09:56:31.178(1)? (STDERR)     at links.js (imports/core/api/links.js:1:1)
W20190315-09:56:31.178(1)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:336:7)
W20190315-09:56:31.179(1)? (STDERR)     at Module.require (packages/modules-runtime.js:238:14)
W20190315-09:56:31.179(1)? (STDERR)     at Module.moduleLink [as link] (/Users/Florian/.meteor/packages/modules/.0.13.0.1aurolz.v3oz++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/reify/lib/runtime/index.js:38:38)
W20190315-09:56:31.179(1)? (STDERR)     at api-server.js (imports/core/api/api-server.js:1:1)
W20190315-09:56:31.196(1)? (STDERR)     at fileEvaluate (packages/modules-runtime.js:336:7)

From what I can see, this.parentCollection.after is undefined, meaning that when you get this._collection, that collection doesn't have hooks on it.

It might just be a question of putting the packages in the right order in .meteor/packages, but I've already tried putting matb33:collection-hooks before grapher, and before this package, without success.

cc: @theodorDiaconu

The underlying meteor-collection-hooks performs two fetch()s on update() for any collection with an after.update hook defined.

• the first to get the ids of all docs matching the selector
• the second to iterate over these docs post-update and fire the after hooks

This can be both slow for large numbers of docs, as well as expensive on memory as it is doing a fetch() of all docs rather than iterating over the cursor.

This denormalize package adds a parentCollection.after.update hook, but also calls parentCollection.update() in any relevant childCollection mutation hook to maintain the caches. The result is that on any child document mutation there is a chain of hooks causing the 2x fetch(), e.g.:

childCollection.update()
  -> childCollection.after.update()
    -> parentCollection.update()
      -> parentCollection.after.update() // 2x fetch()

Related: Meteor-Community-Packages/meteor-collection-hooks#259

Suggestion

Denormalize simply needs to do an parentCollection.updateMany() to update the caches without the extra pre-fetching of ids the hooks do to support arbitrary selectors. Perhaps this package should wrap the Mongo API mutators directly, similar to how the hooks tie in, so as to avoid the chain of hook logic. The one downside I see is exactly that: are there other hooks expected to be chained by the cache update, or even chained denormalization, which this would break? Perhaps this could be an opt-in alternative for maintaining the cache if the user does not require triggering hooks/chaining via the cache update.

I'm almost finished with the integration, however One Meta type of links are not working.

I have the following scenario:
Author -> Profiles
Author Document: profileId: { _id: actualProfileId }

In this case, where nested reference fields are not an array, the cache does not work.
This is the sample config for both:

I20171126-16:08:30.473(2)? cache_authors ::::: { type: 'one',
I20171126-16:08:30.473(2)?   fields: { name: 1 },
I20171126-16:08:30.473(2)?   referenceField: 'profileId:_id',
I20171126-16:08:30.473(2)?   cacheField: 'profileCache',
I20171126-16:08:30.473(2)?   bypassSchema: false }
I20171126-16:08:30.473(2)? cache_author_profiles ::::: { type: 'inverse',
I20171126-16:08:30.473(2)?   fields: { name: 1 },
I20171126-16:08:30.474(2)?   referenceField: 'profileId:_id',
I20171126-16:08:30.474(2)?   cacheField: 'authorCache',
I20171126-16:08:30.474(2)?   bypassSchema: false }

authorCache is empty array
profileCache is missing completely

I tried using autoMigrate, migrate to no avail.

However! The rest of the links one, one inversed, many, many inversed, many meta, many meta inversed have worked flawlessly. Amazing.

Your cache functions has become very big. It's very hard to maintain.

Create a lib folder and split the functionality for many, single, inversed in separate files and functions. Because this way you do the check every time for type on every after update hook.

And if they re-use thing among them modularize those functions too.

Hi Simon,

What is the best way to have many to many relationships? at which point does this cache the doc, on insert?

Thanks,
Chat.

Hi,

In any Meteor version above, but not including 1.6.0.1, I'm getting this error:

https://pastebin.com/raw/CP9Dgep8

Here are the contents of my versions file after updating to 1.7.0.3:

[email protected]
[email protected]
aldeed:[email protected]
aldeed:[email protected]
aldeed:[email protected]
aldeed:[email protected]
aldeed:[email protected]
aldeed:[email protected]
aldeed:[email protected]
aldeed:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
babrahams:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
bredikhin:[email protected]
[email protected]
[email protected]
[email protected]
cfs:[email protected]
[email protected]
chuangbo:[email protected]
[email protected]
cultofcoders:[email protected]
cultofcoders:[email protected]
cultofcoders:[email protected]
daoli:[email protected]
dburles:[email protected]
dburles:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
drewy:[email protected]
drewy:[email protected]
[email protected]
east5th:[email protected]
east5th:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
fortawesome:[email protected]
francocatena:[email protected]
[email protected]
gwendall:[email protected]
herteby:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
jperl:[email protected]
[email protected]
juliancwirko:[email protected]
juliancwirko:[email protected]
kadira:[email protected]
lai:[email protected]_1
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
matb33:[email protected]
mdg:[email protected]
[email protected]
[email protected]
[email protected]
meteorhacks:[email protected]
meteorhacks:[email protected]
meteorhacks:[email protected]
meteorspark:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
meteortoys:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
momentjs:[email protected]
[email protected]
[email protected]
[email protected]
mrt:[email protected]
msavin:[email protected]
msavin:[email protected]
nadeemjq:[email protected]
nadeemjq:[email protected]
[email protected]
[email protected]
[email protected]
ogourment:[email protected]
ongoworks:[email protected]
[email protected]
ostrio:[email protected]
percolate:[email protected]
percolate:[email protected]
[email protected]
raix:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
reywood:[email protected]
[email protected]
rzymek:[email protected]
sacha:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
simple:[email protected]
[email protected]
softwarerero:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
staringatlights:[email protected]
staringatlights:[email protected]
steffo:[email protected]
[email protected]
[email protected]
tap:[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
underscorestring:[email protected]
[email protected]
useraccounts:[email protected]
useraccounts:[email protected]
vazco:[email protected]
[email protected]
[email protected]
yagni:[email protected]
zimme:[email protected]
zimme:[email protected]

I'm seeing this on all version of Meteor after 1.6.0.1. Possibly related to Cult Of Coders packages, however upgrading them doesn't help. (@theodorDiaconu)

What might be happening?