Fix performance issues in collections and subcollections about ampersand-collection

[wraithgar]

Currently the thinking is:

• Make a new base collection module that powers collections and subcollections called indexed-list
• Break subcollections out into two modules ampersand-paged-collection and ampersand-filtered-collection (subcollection can combine these and maintain its current api)

indexed-list will need to maintain both its current named indexing of models (to deal with add/update duplication checks etc) as well as a new offset index (to make indexOf a quick lookup). We need a fast indexOf for most of the things subcollection does (for instance determining if an event needs to bubble up i.e. is this model now a part of the new subcollection). It would still maintain an array of models so that the current exposed array methods would still all work, and also so that subcollections could easily paginate.

There is a branch in subcollection called performance that contains the current wips of getting some of the optimizations ironed out. This branch will likely never be merged in, it's a sandbox for sharing code between those who are working on this problem.

paged-collection would take limit/offset and then slice its parent
filtered-collection would apply a filter to its parent, maintaining parent order unless overridden by its own comparator.

[wraithgar]

indexed-list add would need to use a binary-search + comparator to find the intended index of the new model. Current javascript sorting functions do not allow us to know where our items ended up without a second complete iteration through the list.

This will make all adds with a comparator O(log n) speed (adds without a comparator are simpler of course).

[legastero]

If we have ampersand-paged-collection, we really don't need that to emit anything other than reset events. Something changed? Throw out everything, reslice and emit 'reset'. Any collection view will have the existing item views cached. If it is still slow then, there's really not much we can do. Caveat emptor.

We already made the filtered collection fast in the peformance branch: it was trying to maintain indexes after slicing for paging that was the still-be-solved problem.

[wraithgar]

So performance branch can become ampersand-filtered-collection and then subcollection can intelligently tie them together.

[wraithgar]

Here is paged collection https://github.com/wraithgar/paged-collection

Discussion is now open and welcome for that repo, we can move it to this org as ampersand-paged-collection if we like it.

[wraithgar]

Here is filtered collection https://github.com/wraithgar/filtered-collection

Discussion is now open and welcome for that repo, we can move it to this org as ampersand-filtered-collection if we like it.

[wraithgar]

These two PRs are required for this project AmpersandJS/amp#76 AmpersandJS/amp#74

[wraithgar]

Here are the performance metrics http://artifice.danger.computer/perusinuve.txt

[imlucas]

@wraithgar any updates on this? I've been quite curious about perf & tracking perf over time for ampersand-collection for quite a bit. Has there been any discussion about switching the under-the-hood manipulation to use crossfilter, pourover, ndarray or the like?

[wraithgar]

@imlucas a great deal of effort was put into refactoring subcollection which resulted in https://github.com/AmpersandJS/ampersand-filtered-subcollection and https://github.com/AmpersandJS/ampersand-paginated-subcollection. Currently ampersand-collection doesn't have any plans for more optimization. I'll be closing this as it was primarily about the subcollection performance issues (which were huge). If there is a specific performance problem w/ collection please open a new issue to discuss it since this one's run its course.