Design Proposal: Change pagination_items in returns from lists about server-client-python

[LGraber]

Building a UI for one example.

[LGraber]

You can't remove them. By default, you get 100 items per page and without this return value you won't know if you got all of the items / how many items there are. The only way to "get rid of it" would be if we didn't return a list but instead some more complicated object which, as you called get-next eventually asked for the next page as needed. This, however, is something I would build on top of this low level api.

[LGraber]

to clarify, total is total_available, not how many were returned so it does not necessarily match len(all_workbooks)

[t8y8]

Thanks for your reply.

I didn't realize that .get doesn't get all resources in the collection by default!

I still think it's very weird to expose this implementation detail do the high-level get call.
When 99% of the time folks are ignoring a return value maybe we don't need to return it.

But your comments make me wonder if there's a different route for me to propose -- should .get by default get all resources. That's a different issue that I can file.

For now, what about just reversing the return order?

all_workbooks, _ = server.workbooks.get()

[t8y8]

I've got a working decorator.
It obeys the request options if you specify them, else it just loops at the default size.

I can do a bit more to make this happen by default, but can disable it if you'd like after I learn more about decorator patterns in Python.

I'll get some working code up for a PR with a test but I need to learn how to use the test framework first.

Here's a preview

def paginate(get_call):
    from math import ceil
    from ..request_options import RequestOptions

    def wrapper(self, opts):

        all_things = []

        # First call
        print("FIRST CALL")
        res, PI = get_call(self, opts)
        all_things.extend(res)
        print("ALL THINGS HAS: " + str(len(all_things)))

        # Build loop Conditions
        number_of_workbooks = PI.total_available
        pageSize = PI.page_size
        print("TOTAL AVAILABLE: " + str(number_of_workbooks))
        print("PAGESIZE: " + str(pageSize))

        # Do it
        if number_of_workbooks > pageSize:  # Need to page through responses
            print("NEED TO LOOP")
            number_of_pages = ceil(number_of_workbooks / pageSize)
            print("THERE ARE THIS MANY PAGES: " + str(number_of_pages))
            for pageNum in range(2, number_of_pages + 1):
                opts = RequestOptions(pageNum, pageSize)
                res, _ = get_call(self, opts)
                all_things.extend(res)

        return all_things

    return wrapper

Then the get call on the workbooks_endpoint just becomes:

    @paginate
    def get(self, req_options=None):
        logger.info('Querying all workbooks on site')
        url = self._construct_url()
        server_response = self.get_request(url, req_options)
        pagination_item = PaginationItem.from_response(server_response.content)
        all_workbook_items = WorkbookItem.from_response(server_response.content)
        return all_workbook_items, pagination_item

/cc @RussTheAerialist @shinchris @LGraber

[LGraber]

Couple of comments:

1. I was thinking that I wanted a class which could be used to "stream" the list back and hide the pagination, but not necessarily get them all at once. This is a layer on top of what we have and similar to yours above, is only really one class, but this one exposes the list and as you iterate, if it runs out but knows that there are more on the server, it will go and get them. It is useful for building "infinite" scroll experiences. This class could be used to preload everything if we wanted, also. I am not saying we have to do one or the other and not both but we should consider all of the use cases.
2. Your decorator always pulls all items which is not something our library should do by default. If we want to key this off of them setting the page_size to "ALL" or something like that, okay, but we should not enforce a behavior that is not always desirable
3. There are actually better ways to do "ALL". Specifically, I think a larger page size (perhaps the max of 1000) would be advisable. if they indicate "ALL" we should probably be increasing the page size to try and optimize the interaction with the server. Network latency is not an issue since they will block until we get all of them anyways.

[t8y8]

Definitely a valid approach, and we could do one or both.

But I really don't see the use case for not wanting them all by default, or with lazy loading them, as far as the complexity it could introduce into the code.

When getting a list of all workbooks you're likely going to do something to that list shortly after, either a bulk update or a bulk filter/map/read type operation. I don't think we save much time by calling them lazily if most code will be:

wbs = server.workbooks.get()
for w in wbs:
    do_stuff(w)

We have other methods to get a specific workbook (get_workbook_by_id). I think this endpoint really implies 'all'. Maybe that's the philosophical question to resolve.

Consider the case where I build a RequestOptions with a filter and a sort and forget to change the default page size, I don't realize my 100 results aren't all of them, without all my calling code checking total_available. But why would I ever not want all my search results?

It's possible I just can't think of the use cases around this -- anyone have any that I'm missing?

[graysonarts]

@t8y8 Are you content with the implementation of the Pager object wrt to this Design Proposal? If so, can you close this issue?

[t8y8]

Yup, it’s been working smoothly for months. I didn’t address tags or the need to copy objects but I think paging is much saner now, I’m happy.