Massively corrupted nested set about awesome_nested_set

[brendon]

Actually, just noticed some notes about concurrency in another nested set plugin. Does awesome nested set handle concurrent updates properly or will the database become corrupt when two people do changes that require a lot of updates to left and right at the same time?

[danielmorrison]

Brendon, I'm not sure what caused the corruption. Maybe a non-transactional database? It shouldn't get corrupted in normal use.

Regardless, calling rebuild! will get you back to sanity. That's what its there for. (actually, I use it to convert from acts_as_tree or other systems).

[danielmorrison]

Concurrency is really up to the underlying database. You can always wrap nested set operations in a transaction block if you're having situations like this.

[brendon]

Thanks Daniel, yep rebuild should do the trick, but unfortunately it's a CMS and their pages etc.. will fall out of order. Hopefully ordering by lft should mitigate this a wee bit :)

I'm using MySQL so I think transactions aren't the problem. Do transactions prevent concurrency problems?

[danielmorrison]

I guess you could have issues with multiple inserts, that would still work with transactions, but only the lft and rgt would be messed up, the parent would still be fine. Table locking might be your friend, or you can patch up the plugin to be much more crazy. :/

[brendon]

Thanks Daniel, after much testing, you're correct. In my app there is an easy way to add many children in quick succession. This easily corrupts the lefts and rights. I've looked at the mysql setting: 'transaction-isolation = READ-COMMITTED' but to no avail. It still seems to happen. I guess the only way is to block the whole table from updates when an update is occurring. Are you aware of any way to do this effectively at the database level?

[brendon]

Oh and the trouble with the broken lefts and rights is we have dependent destroy turned on, so people destroyed bits of their site that they didn't intend to. A much more dangerous problem under the surface :)

[aquabu]

Brendon, I haven't looked at this in a while but I was seeing some serious corruption issues a while back and solved them by: 1) using scopes for nested sets 2) setting mysql transaction isolation level to serializable 3) using transaction blocks and in some cases (e.g. moving nodes) selecting the other nodes in the same scope with ActiveRecord#find :lock => true. This is paranoid and slow but much better than manually rewriting the bounds of corrupted nested sets - which unfortunately we had to do before adopting this approach. I think these issues may show up when there are multiple transactions operating on different parts of the tree - if you have scopes you can lock just the nested set that you are updating.

[brendon]

Thanks :) I ended up switching to the ancestry gem (much easier to grapple with). I tried locking the whole table for the left/right update part of the gem, but that even didn't stop the corruption. We only have one tree that governs the whole site so unfortunately scoping was out of the quesiton. Ancestry seems to have worked out well, and I combined that with acts_as_list for ordering support. Ancestry also has bonus depth caching support so that'll be cool to use to help us improve the speed of our index_bar generation stuff. :) Thanks for your tips though :)

[galori]

aquabu - can you expand on #1 - how do you use scopes for a nested set update? Specifically if I am creating a new record and then calling move_to_child_of, where does the scope play a part?

We're also experiencing nested set corruption, will try to follow your suggestions and see if it addresses the issue.

[brendon]

Hi Galori, our app uses a seperate database for each of our customers (so we have like 100 databases). I'm not 100% sure, but I noticed the other day that some of our older customers who were migrated from another server had a mix of MyISAM and InnoDB tables in their database. It's my understanding that MyISAM tables don't support transactions, so that would make it easy to corrupt those tables. Perhaps double check that you're using InnoDB tables (if you're using MySQL)?

I say I'm not sure, because I think I noticed corruption even on InnoDB tables.

As great as awesome was, I've found the move to ancestry + acts_as_tree to be quite comfortable. I've posted some special methods (like move_to_child_of) etc... in the ancestry wiki that should make the migration much easier.

[xaviershay]

Assuming the code itself is solid, set your transaction isolation level to SERIALIZABLE. Locking all the rows may work in MySQL, but that's just getting lucky and not because it should work (because of it's gap/next-key locking). It's unnecessary under serializable.

A transaction by itself is not sufficient.

Background (acts_as_list has the same problem):
http://rhnh.net/2010/06/30/acts-as-list-will-break-in-production
http://www.engineyard.com/blog/2010/a-gentle-introduction-to-isolation-levels/

[MarkusQ]

Setting transactions to serializable is not sufficient. It simply converts some instances of the race condition from a corruption to a deadlock (which will either timeout or raise an exception). It also imposes an excessive overhead (effectively acting as a whole-table lock).

I believe the correct fix is targeted row locking. See: https://github.com/MarkusQ/awesome_nested_set for a fork with my proposed fix (based off of vamosa's).

[brendon]

Sounds cool :D I'm not sure whether or not I should close this case? I guess until a fix is pulled through, it's still a valid ticket? :)

[parndt]

If a fix is possible, it'd be nice to have it (must pass tests)..

[parndt]

@MarkusQ was this solved by your recent merged pull request?

[MarkusQ]

I believe so, yes; at least 1) we were able to reliably reproduce these symptoms with the unpatched version; 2) we were able to show (with logs & whiteboard work) how the problem could happen, 3) the patch fixes the hypothesized code paths, 4) we can not reproduce the problem with the patched version (which we are running in production on a moderately heavily trafficked application).

[parndt]

Awesome, thanks!