Forced paging breaks service about w3c-api

[vivienlacourba]

@darobin, there is now a this now a limit for the maximum number of items for pages using pagination. And as you discovered it is set to 100.
This limit was added as some request can be quite intensive and we faced server memory limit issues on some big lists. FYI, our implementation is similar Github's pagination and we used the same limits.

As discussed with you and @jean-gui on July 17th as a temporary solution we increased this limit to 500 so that it does not break your app during your vacations but please work on traversing paginated results when you comes back as we want to set this max limit back to 100 items before releasing the API.

[darobin]

I have now made the repository manager use the new w3capi library that just slurps pages in for it. You can revert the temporary solution, thanks!

I am, however, seeing some low performance. It's not really an issue for the tool, at least not at this point, but the simplest request for just a single item never seems to take less than 500ms (I tried from several machines).

[jean-gui]

I think that is due to caching not being fresh enough because of the low number of requests. All responses are cached for 900s after which they have to be revalidated. After these 900s are over, there is a grace period of 10s during which if a request to the stale resource is made, it will be served while a newer version is fetched from a backend for subsequent requests. So data can be stale for at most 910s.

These values probably need to be adjusted, mostly the grace period. We could set it to a very high value (not sure what would be considered reasonable or not) - let's say 1 day for the sake of example - but in this case, if the resource is not accessed often, clients might received data that has been stale for 1 day and 15 minutes.

Another idea would be to run a cronjob on the server that would automatically refresh the cache every once in a while.

Finally, we have some ideas to improve performances of backend servers, we will work on that after the first version of the API is released.

[darobin]

I've played with this but it looks like caching is only part of the story. Even if I run my test suite multiple times (which always calls the same objects) I can't get the request times to fall under ~500ms, even for basic stuff like getting the JSON for one group with no embedding. Copying the static JSON to my own server (which is just a cheap off-the-shelf VM somewhere on the East Coast) and using the same code to hit it I get results that are 3x faster.

Obviously there are plenty of factors that could be playing in here, but 500+ms for a cached resource that's ~600 bytes strikes me as likely to indicate an issue, which is why I'm raising it :)

[vivienlacourba]

@darobin we have reverted the pagination limit to 100 items so I'm closing this issue. I'll open a new one regarding your comment on low performances.