Improve get plans query prefetching about lotus

[genego-io]

I led a short investigation into this performance issue. The optimization likely would come from looking more closely at PlanSerializer, which now initializes the PlanDetailSerializer. This is done with the many=True flag, but regardless still calls the Query for each object in the returned list.

@diego-escobedo, I am not seeing 180 db hits. How many plans do you currently have that are being loaded?

For example, if I set up 3 plans (which only have a name), and then call /app/plans/:

If I dig a little deeper, I arrive at PlanDetailSerializer.get_versions being responsible for 39 of these queries (13 for each nth Plan).

    def get_versions(self, obj) -> PlanVersionDetailSerializer(many=True):
        return PlanVersionDetailSerializer(
            obj.versions.all().order_by("version"), many=True
        ).data

If I remove get_versions, then obv. we have a much more performant request, the time is almost cut in half:

Ideally, this could be annotated and combined into a join with 1 or 2 queries.

---

Additionally, for every request to /app/plans/ there is another call being made to /app/organizations/, which unless I am mistaking, should not be necessary, but it is resulting in another 33 additional queries.

Can't solve the issue, but perhaps its enough help to investigate.

If I have some more time I will look into it as well.

[diego-escobedo]

Seeing the number of queries here. I have 7 plan versions in mine so its possible its doing even more inefficiently in the get_versions. First step I'm thinking of taking is using the to_attr in the prefetching of the versions so that it gets used explicitly in get_versions. I suspect calling obj.versions.all().order_by("version") refetches everything.

this cuts maybe like 7 queries per plan. Another thing I'm noticing is I'm not prefetching recurring charges, which were separated from being columns of plan versions into their own models.

I'm curious at this point to see how much of the remaining queries are from stuff further down the stack. I prefetch everything we should need in the get_queryset method of the PlanViewSet, but it seems likely most of this ends up not being used. I suspect part of this comes from display_version, which is just one of the versions but gets pulled directly by the PlanDetailSerializer from the model instead of using the already prefetched version.

Next problem seems to be these queries associated with the recurring charges.

Started using the prefetched versions instead of just whatever the plan decided to pull,

Now need to figure out why that single plan also gets pulled, and where these organization queries are coming from.

Turns out they were coming from the TimezoneFieldMixin. It was 1. using db lookups instead of caching (and deleting cached timezones when changed) and 2. checking even when there was no datetimefield in the representation.

getting close! just weird that we have another query lookup after the entrypoint post prefetching.

boom! demolished those queries. Thank you so much for the help @genego-io