Performance of serializing nested collections is poor about marshmallow

[sloria]

@mgd722 I haven't compared the two usages in a while, but they should be similar. If you're serializing ORM objects, I'd first look into your relationship loading technique and make sure you're not running into the n+1 problem.

[sloria]

The deepcopy operation is expensive, but necessary, so that serializers can store errors from nested serializers.

I did a little work with cProfile and your code above (the gist of the script is here) and found 2 significant speedups:

• Passing an instance--not a class--into a Nested field.

Example:

collaborators = fields.Nested(UserSerializer(), many=True)

instead of

collaborators = fields.Nested(UserSerializer, many=True)

This avoids repeating the initialization code (including the deepcopy) for each collaborator. In the future, it'll be better to cache the nested serializer object, or disallow passing classes altogether.

• Overriding __deepcopy__ method of field objects so they are only shallow copied (9c0f062). Even though the declared_fields dictionary must be deep-copied, field objects themselves don't need to be deep-copied.

These two modifications decreased the execution time of the above script by almost half.

Thanks for reporting this. I will continue to do more profiling and see where performance can be improved even further.

[sloria]

I underestimated the effect of passing in an instance into a nested Field: doing this for both the "user" and the "collaborators" field reduces the total runtime of the profiling script from ~5.7s to ~1.6s.

class BlogSerializer(Serializer):
    title = fields.String()
    user = fields.Nested(UserSerializer())
    collaborators = fields.Nested(UserSerializer(), many=True)
    categories = fields.List(fields.String)
    id = fields.String()

[sloat]

That's great! Very interesting...

Thanks again!

[sloria]

Glad I could help.

I've made some further improvements so that performance should be good whether you pass in a Serializer class or a Serializer object into a Nested field.

[andras-tim]

I have same issue with serialize, and I was run into this issue.

I think, you should update the documentation http://marshmallow.readthedocs.org/en/latest/nesting.html about use instance instead of class passing trough.

[mgd722]

@sloria, just to be clear-- performance should now be similar in both of the following scenarios:

fields.Nested(UserSerializer(), many=True)

fields.Nested(UserSerializer, many=True)

so if nested fields are running slow for me it's just my shitty code?