ampersandjs / ampersand-collection Goto Github PK

Currently collection.indexOf and everything about subcollections is very inefficient. Normally you don't see this problem in collections (indexOf isn't used on orders of magnitude that cause problems for most people), however in subcollections the problem becomes quickly untenable for collections even nearing a hundred items.

After a lot of hacking and testing, I (with a lot of help from @legastero and @latentflip) have decided that a new approach is needed. This issue is for outlining that approach, with hopefully minimal feedback. I would kindly ask that until we have an actual proposal ironed out we keep debate and discussion to a minimum. (i.e. this has to get to even a certain point before we're ready to enter a wider discussion).

This issue will be mostly about visibility with the @AmpersandJS/core-team as we work through the problem.

If you would like to see a quick demo of the problem, run this artifice.danger.computer/ubavuzoxek.js but note that the devDependency for ampersand-model is [email protected] so you'll have to update that to remove isNew errors.

I'm going to try to get a test case this weekend, but I don't know if I will be able to reproduce the environment.

My karma/jasmine unit tests pass with ampersand-collection 1.4.2 but time out 100% of the time when using 1.4.3. I think that the reindex code is hitting an infinite loop or is causing too many cascading operations.

When I change a property defined by the comparator the collection does not sort itself. Is this intended?

var Model = require('ampersand-model');
var Collection = require('ampersand-collection').extend({model: Model, comparator: 'attribute'});


var models = [new Model, new Model, new Model];
var collection = new Collection();
collection.reset(models);
console.log(models.map(function(m){return m.cid}));
console.log(collection.models.map(function(m){return m.cid}));

results:
['state1','state2','state3']
['state3', 'state2', 'state1']

Models will get sorted in reverse order when the comparator attribute is undefined on a model.

This result can be replicated in chrome and firefox but not phantom js.

This works correctly in Backbone.Collection as it uses the underscore sortBy method and not Array.prototype.sort .

Swapping these two lines https://github.com/AmpersandJS/ampersand-collection/blob/v1.3.17/ampersand-collection.js#L227-228 (and https://github.com/AmpersandJS/ampersand-collection/blob/v1.3.17/ampersand-collection.js#L235-236) seems to do the trick for chrome and firefox but breaks phantom js.

ampersand-subcollection and ampersand-collection both use different sort methods.

Ampersand-subcollection uses lodash sortBy, and ampersand-collection uses native array sort. I ran into an issue where I was using the wrong sort method.

Not sure what the best change to make is, but maybe editing docs to make it more clear that collection/sub-collection have different APIs or normalizing the API to use the same sort method.

https://github.com/AmpersandJS/ampersand-subcollection/blob/master/ampersand-subcollection.js#L175

https://github.com/AmpersandJS/ampersand-collection/blob/master/ampersand-collection.js#L238

I'm running into the following problem: what if, between when I send a fetch call, and when the call returns, something happens in my app that makes me want to cancel the fetch. For example, if a user tries to load more entries in a list, but then, before that XHR response is received, decides to just refresh the list wholesale, I would like to cancel the results of the first call. To be more clear:

1. User presses "see more items" button in list, triggering myCollection.loadNextPageOfResults()
2. User gets impatient, and decides to refresh the whole page
3. For some reason, the API call from #2 returns first.
4. Finally, the API call from #1 returns, but since we have no way to cancel it, it appends the results of myCollection.loadNextPageOfResults() onto the list, even though that is not necessarily desired

What I'd like to implement is a .cancel() option that returns a boolean. If it returns true, then the entire fetch operation is aborted before and set/reset of the actual collection can occur - it will be as though the fetch was never sent to begin with. This could optionally trigger some sort of event as well, if its deemed worthwhile. I think this would be a bit less confusing than the currently undocumented .set property.

Basically, instead of having, the .set conditionals on this line and this line, I would just wrap lines 17-19 in something like if (!opts.cancel || !opts.cancel(self, resp, options)) {}. Thoughts?

It would be nice to have a chance before a model is actually inserted/removed from a collection or changed to enforce any collection invariants.

For example, say I have a collection of apple models, but I am only allowed to have one green apple at a time. On a pre-add I could check that the new apple is green and remove any existing green apple, and this would work regardless of what actually called the .add().

Likewise, if we had a pre-change event from ampersand-state, then the invariant could still be enforced when an apple changed from red to green, no matter where or what triggered the change.

Maybe a better example would be an invariant that there can be only a single model where selected is true.

(any pre-change event would be a matter for ampersand-state to implement, of course)

I would expect the following code to result in a totally empty collection:

var PersonCollection = require('./models/person-collection');
var collection = new PersonCollection();

Instead, the collection contains one item and collection.length === 1. This makes for some confusing UI defects when a collection is tied to subviews.

My expectation is that a new collection would be totally empty until I use REST to fetch some data or add items in some other way.

Currently it's impossible to iterate over collection via for...of from ES2015 (some sort of Symbol iterable bug). Workaround is to iterate over collection.models.

There is a whole slew of problems once you add an element with id:0.
For example, if you keep performing collection.set(collection.serialize()) (which should be more or less idempotent) you get more and more elements appended to the end, all of which have the same .cid.
Moreover you can not clean up this collection as collection.set([]) is unable to remove them.

I think the problem is in this line:

if (order && ((model.isNew && model.isNew() || !model[this.mainIndex]) || !modelMap[model.cid || model[this.mainIndex]])) order.push(model);

In particular the condition !model[this.mainIndex] treats id=0 as a missing id, and subsequently schedulles the model for ordering phase.
In that phase it gets pushed at the end:

var orderedModels = order || toAdd;
for (i = 0, length = orderedModels.length; i < length; i++) {
   this.models.push(orderedModels[i]);
}

It's also important that the model pushed to the order array is actually the already existing one found by get(0), which means that we push exactly the same instance at the end of this.models array, so now we have two entries with the same .cid.

This in turn means that this elements never get purged, even if one calls set([]), because of two other problems (I think there is a separate bug for that in issue tracker) - one is that there is

if (query == null) return;

in get(query) function, and for some reason the toRemove array contains undefined as values.
The second problem is that once any element with given cid is removed it is also removed from _indexes.cid. So if there are multiple such elements, once the first is removed, then subsequent are irremovable, due to:

 model = models[i] = this.get(models[i]);
            if (!model) continue;

in remove().
I think the two problems are related in that, once there is an element missing from the index, it is quite possible that undefined will occur in the toRemove array.

In general, I'd suggest scanning the whole code against expressions like "!x" and carefully replacing them with more semantically valid x === undefined or blah in x or whatever the programmer actually had in mind.

When using ampersand collection with a model with a non-standard idAttribute set the collection will allow the same item to be added multiple times.

This is at least partly due to the code assuming the id attribute will be id for example https://github.com/AmpersandJS/ampersand-collection/blob/master/ampersand-collection.js#L162

It's not entirely clear to me what the "right" solution is hear due to the interactions between idAttribute in the model and the mainIndex attribute on the collection.

Just ran across this one, and thought I'd share in case others are searching. Could be easily fixable, but I could understand why it wouldn't be.

On this line the native .bind is being used to bind the comparator to this. This only happens on a multi-argument comparators: comparator : function(a, b){}.

This causes phantom js (using ~1.9.7-1) to fail unfortunately. While that's not really ampersand's concern, this could be modified to use lodash .bind or amp .bind. This seems to be the approach in ampersand-state where underscore.bind is used.

I'd be happy to submit a PR to fix, using amp .bind if desired.

We were able to fix by loading es5.shim in phantom for those who run into similar issue.

fetch, create, sync existing in ampersand-collection-rest-mixin (as opposed to it living in the collection itself as in backbone), when wanting to use different sync strategies it would make sense to have a generic mixin wrapper you could inject sync strategy into or to just include in ampersand-collection to keep it consistent with ampersand-model

I will admit I'm not 100% sure the source of this issue yet. I'm still trying to pinpoint the source, but there is definitely an issue so I figured I'd get this logged.

Here's an example of my data model:
row.js
module.exports = AmpersandModel.extend({ collections: { columns: Cells } });
cells.js
module.exports = AmpersandCollection.extend({ indexes: ['columnDefinitionId'], model: Cell, });
cell.js
module.exports = AmpersandModel.extend({ props: { id: ['number', false, null], columnDefinitionId: ['number', false, null], value: ['string', false, null], } });

When I call row.save(...) I'm seeing issues with the indexes of my cells collection. The indexes look good before, but not after collection.set(...) is invoked.

1. at some point, _reset is invoked, and at this point, indexes is no longer what I defined. It is an array (length 500, which seems absurdly large for a collection of ~5-6) that looks something like this:
   
2. once set has completed, my collection's _indexes looks like this:
   

As you can see, the cid and id indexes are rebuilt, but the columnDefinitionId index that I defined in my collection is not.

1. I've found this happens on save, but not fetch.
2. _addReference appears to be working properly, and, at the point this is invoked for each new model, the new cells are indexed properly.

Any help or insight here would be appreciated as this is creating a pretty major blocker for my project right now. Thanks!

Backbone added an "update" event in 1.2.0, which is convenient when binding to React, where we typically don't want to rerender for each individual "add" event, but just once, at the end.

In the documentation it states that instead of a Model you can pass a function to the model property of a collection that returns the new instance, in order to have polymorphic behaviour. While it's a great feature, it's actually broken right now.

See this example

The _prepareModel method that's called when adding new models call isModel in order to find out if a new instance should be created. However, isModel, as I know it from Backbone, does a direct instanceof check with the property of model, which in case of the polymorphic function fails because it's just a function.

I would post a PR if I had a clear idea on how to fix it. The problem I face is finding a reliant way to figure out whether the function in model is just a normal function or a constructor for a model. Suggestions anyone?

this is a function that ampersand-view has which could probably be used elsewhere too, starting here

When initializing a collection from a multidimensional json, any sub collections will have an empty model as its first value in the sub collection. An empty collection will even contain 1 empty object in the collection.

For example,

var schools = [{
"school_name" : "A",
"students" : []
},{
"school_name" : "B"
"students" : [{"name" : "Lance"}]
}];

new SchoolCollection(schools);

Will result in the student collection having an empty object for school A.
School B's student collection will contain an empty object and the object for Lance.

What events are emitted by a collection, and when.

There are some spots where .id is still used, even if the model has specified a different idAttribute field.

It'd be nice if tests could be run like this: browserify test/* | tape-run | tap-spec, but they fail on the isCollection immutable test because of this phantomjs bug: ariya/phantomjs#11856

Not high priority, but just a marker for when phantomjs fixes that bug.

I have an ampersand-rest-collection object definition that has been extended with a comparator function as follows:

var Collection  = require('ampersand-rest-collection');
var EventDate   = require('./event-date');

module.exports = Collection.extend({
    model: EventDate,
    url: '/api/event-dates',

    comparator: function (model) {
        return model.eventDate.getTime();
    }
});

However, when I instantiate the collection and fetch data from my API endpoint, the data is sorted, but not always correctly. Even when the data is sorted correctly the first time, if I then re-fetch with different data parameters (thus changing my result set), the sorting breaks completely.

I've tried the following versions of the comparator as well:

comparator: function (model) {
        return model.eventDate;
    }

comparator: function (m1, m2) {
        return ((m1.eventDate > m2.eventDate) ?
            1 : ((m1.eventDate < m2.eventDate) ? -1 : 0));
    }

I've also tried removing the comparator function altogether and passing the following options object when newing up the collection:

{
    comparator: 'eventDate'
}

All of my attempts seem to yield the same result.

Has anyone else experienced this issue? Or am I misunderstanding how the comparator function works?

Thanks,
Genaro

https://github.com/AmpersandJS/ampersand-collection/blob/master/ampersand-collection.js#L310-L311

Pretty sure we want to send the old values to _deIndex

    _onModelEvent: function (event, model, emitter, options) {
        if ((event === 'add' || event === 'remove') && emitter !== this) return;
        if (event === 'destroy') this.remove(model, options);
        if (model && event === 'change:' + this.mainIndex) {
            this._deIndex(emitter);
            this._index(model);
        }
        this.trigger.apply(this, arguments);
    }

A collection emits events that are bubbled up from the models it contains. This is the current behavior and expectation from even pre-ampersand (i.e. backbone).

It would be nice if there were a way to event on collection state events, specifically the length. Unfortunately you can't just start emitting change:length events because everything listening to events coming out of the collection would have to start paying attention to the type of object emitting the change event, because it could now either be the collection itself or a model it contains.

There are two ways to potentially solve this.

• emit collection state events from a different namespace, i.e. instead of change collectionChange
• attach a state object to the collection that will hold and emit these collection state attributes somewhere like at collection.state

The latter feels more properly separated but this is something a lot of people have asked for (i.e. being able to derive an empty attribute) and I don't see any discussion happening lately so I'd like to get this out there.

collection.add([
  { id: 0, value: "zero" },
  { id: 1, value: "one" }
]);

collection.get(0); // expected { id: 0, value: "zero" }, but returns `undefined`

I think this is because of the check for !query on line 162 here:

get: function (query, indexName) {
    if (!query) return;
    var index = this._indexes[indexName || this.mainIndex];
    return index[query] || index[query[this.mainIndex]] || this._indexes.cid[query] || this._indexes.cid[query.cid];
},

I thought this was surprising. Is it intentional?

Just noticed yesterday it's not listed.

If I pass a collection to a template, that template should be able to iterate through the collection (using for, for example) and get back the models in the collection. Currently you cannot pass a collection to a template and have it enumerate like a plain array would be expected to.

Would you support a PR for unshift?

I understand it's just collection.add(obj, {at: 0}) but I expected unshift to exist since backbone has it

At Remix we use a lot of geographical data, which we need to index using a spatial index, for which we use rbush. Therefore we need to insert models into rbush any time they are inserted into a collection, and removed from rbush when they are removed from the collection.

My first idea was to listen to add and remove events, but unfortunately they can be silenced. So the only option left is to override methods. It turns out that set calls remove, so overriding remove works for that case. However, set doesn't call add, but add calls set (which is inconsistent/confusing in itself). So I end up having to do something like this:

set(...args) {
    // Store what models we have before calling `set`
    const previousModelIds = this.map(model => model.id);

    // Call super.set()
    const modelArrayOrObj = Collection.prototype.set.apply(this, args);

    // See which models have been added
    const models = Array.isArray(modelArrayOrObj) ? modelArrayOrObj : [modelArrayOrObj];
    const addedModels = models.filter(model => !previousModelIds.includes(model.id));

    // Load new models into rbush
    const treeNodes = addedModels.map(createTreeNode);
    this.tree.load(treeNodes);

    // Return what was originally returned by `super.set()`
    return modelArrayOrObj;
},

Am I missing something here? Or is this actually so inconvenient?

There are a couple of ways I see how this can be fixed. The nicest one IMO would be to have set call add, to be consistent with remove. But I can see how that would be a tricky change. Another option would be to expose this.addedModels, this.removedModels, and this.mergedModels, after calling set, and have remove call set (for consistency). What do you think?

I need to use this as ampersand Model and collections.

[ // Collection1
  [ // ItemsCollection
    {}, //ItemModel
    {}
  ],
  [
    {},
    {}
  ]
]

Can I just use ? Is this a proper way to do it ?

Collection.extend({
  model: ItemsCollection
});

With Backbone, it's possible to bind this in a collection helper:

collection.each(function(model, index) {
  ..
}, this);

With Ampersand, it's not:

collection.each(function(model, index) {
  ..
}.bind(this));

May it be added as well?

Was wondering if there was a way to have properties (props, derived, session) for ampersand-collection. Would be particularly useful to have a derived properties that can return values based on the collection's array.

For example, say you have a collection called products filled with product models. Each product model has a price property. It would be nice to define a derived property on products that could map/reduce the product prices into a total property. This total would trigger event listeners bound to it (as ampersand-state does with its properties).

Is this possible?

I've just jumped in Ampersand and I really like it.

One thing that I miss is the ability to retrieve more than one method using an index.

For example retrieving all the models that match a query not a single one.

I gave a look to the code and would like to make a pull request but first wanted to hear what do you about it

const State = require('ampersand-state');
const Collection = require('ampersand-collection');

const StateCollection = Collection.extend({});

const Country = State.extend({
  collections: {
    states: StateCollection,
  },
});

const ugh = new Country({
  states: null,
});

console.log(ugh.states.length, ugh.states.models.length);

const derp = new Country({
  states: [],
});

console.log(derp.states.length, derp.states.models.length);

Output is 1 1 and then 0 0. Length should be zero in both cases, correct?

Currently working on this. Expect to be done Jan 18th 2015.

The docs are missing a section for comparator; it wasn't immediately obvious to me that I could set the comparator in an .extend() call, or especially that I could use a string as the comparator to use a property on the model for sorting.

Of course, knowing Backbone I assumed this would be the case, but Backbone knowledge shouldn't be a prerequisite here.

Collection.sort() breaks when comparator is just a string since this.sortBy() is not defined.

If the collection is created specifying a model class, the mainIndex should be set to the model's idAttribute