Groot for Large Data Sets about groot

[aspcartman]

For large datasets it's always recommended to do things the hard way: by hand. Universality of tools comes in price. Also you should consider using "background" contexts.

[o15a3d4l11s2]

I am also interested in possible techniques for speeding up the persistence process. I tried resetting the context before persisting entities, but this did not affect the speed.

[gonzalezreal]

I think the performance problem resides on the structure of the data, rather on the amount of data. Of course this becomes more evident on large datasets.

One thing that affects performance when serializing from JSON is object uniquing, as it requires fetching data from the database before inserting.

If you take a look on how Groot is implemented, there are three serialization strategies:

• Insert
• Uniquing
• Composite Uniquing

As you may guess, the first one is the most performant as it does not fetch from the database. If you know that there is no duplicate data in your data set, DO NOT set identityAttributes in your entity. This will make Groot use the Insert strategy.

Groot will pick the Uniquing strategy if the identityAttributes annotation has a single attribute, otherwise it will pick the Composite Uniquing strategy.

The Uniquing strategy requires one fetch for every array of JSON objects, whereas the Composite Uniquing strategy requires one fetch for every single JSON object (it is potentially the slowest of the three strategies).

I hope this sheds some light on the subject.