[0.18.0] Flop.all applies a default limit of 50 about flop

[woylie]

I need to think through this again. I'll release 0.18.1 with the changes tomorrow.

[woylie]

You can now do this:

case MyFlop.validate(flop, for: Entity, default_limit: false) do

[woylie]

That might have been a rash decision. For context, how do you use Flop? Would your case be solved if you set a different default limit by adding it to your config or schema or by passing it all? Or do you really run boundless queries with Flop, without any pagination? Does the default max limit cause any issues for you as well?

[lucacorti]

I run queries where I want all results for a specific set of filters. So yes, these are boundless but in a controlled way. I am using cursor pagination and have a configured default (25) and max (100) limit, but apparently only limit: 50 appears in the Flop struct.

Again, this might be a matter of expectations. I expected all to be able to give me all results for a specific query and not use pagination, which is what it did before 0.18.0.

[woylie]

I am using cursor pagination and have a configured default (25) and max (100) limit, but apparently only limit: 50 appears in the Flop struct.

That's weird, the default limit that you configure should override the global one. Where did you define it, in a config module (use Flop) or in the application environment?

[lucacorti]

In a config module MyFlop with use Flop. I then call Flop.validate and MyFlop.all.

[woylie]

Ok, that makes sense. Can you try the branch add-validate-to-config-mods and change Flop.validate to MyFlop.validate?

[lucacorti]

On that branch I get the following:

1. MyFlop.all returns 26 schemas out of 100 so it seems like it is now kinda obeying the configured limit (+1 ?).
2. MyFlop.validate has started returning errors saying order_by is required where it previously worked.

[woylie]

all returns the limit + 1 when cursor pagination is used, so that Flop can set the has_next_cursor value correctly. The extra entry is removed by run (run calls all internally).

Cursor pagination doesn't make sense without a defined order, and the cursor is based on the order field values, so you either have to define a default order on the schema, pass a default order to the validate function, or set the order in the parameters. But this is not new. Also, to use cursor pagination, the first or the last parameter or a default limit must be set, and that's also not new. So if you previously successfully ran a query without a limit and/or without order_by, no cursor pagination was involved in that.

[lucacorti]

Ok, so regarding my use case, how should I go about fetching all schemas regardless of pagination?

WRT order_by, I failed to mention, a default order is defined on the schema, but apparently is not taken into account anymore by MyFlop.validate while it was in 0.17.2.

[woylie]

Ok, so regarding my use case, how should I go about fetching all schemas regardless of pagination?

Would it work for you to pass a very high default limit to the validate function, like 9_999_999, or whatever row count you never would expect?

WRT order_by, I failed to mention, a default order is defined on the schema, but apparently is not taken into account anymore by MyFlop.validate while it was in 0.17.2.

There is a test for cursor pagination with a default order already. I'd need steps to reproduce this.

https://github.com/woylie/flop/blob/main/test/flop/validation_test.exs#L366

[lucacorti]

Ok, so regarding my use case, how should I go about fetching all schemas regardless of pagination?
Would it work for you to pass a very high default limit to the validate function, like 9_999_999, or whatever row count you never would expect?

Yes, also, I think I can use MyFlop.filter and MyFlop.order_by to build a query and then pass it to ecto to perform the query without pagination like this

query
|> MyFlop.filter(flop)
|> MyFlop.order_by(flop)
|> MyRepo.all()

There is a test for cursor pagination with a default order already. I'd need steps to reproduce this.
https://github.com/woylie/flop/blob/main/test/flop/validation_test.exs#L366

Switching from Flop.validate to MyFlop.validate triggers the issue for me.

defmodule MyFlop do
  use Flop,
    repo: MyRepo,
    default_limit: 25,
    default_pagination_type: :first,
    max_limit: 100,
    pagination_types: [:first, :last]
end

defmodule MySchema do
...
  use Entity,
    filterable: [:id, :company_id, :payment_date, :expected_payment_date],
    sortable: [:created_at],
    default_order: %{
      order_by: [:created_at],
      order_directions: [:desc]
    }
...
end

This works fine:

case Flop.validate(flop) do
  {:ok, flop} ->
    MyFlop.count(entity, flop, for: entity)

  {:error, %Meta{errors: errors}} ->
    {:error, errors}
end

This returns {:error, [order_by: [{"can't be blank", [validation: :required]}]]}:

case MyFlop.validate(flop) do
  {:ok, flop} ->
    MyFlop.count(entity, flop, for: entity)

  {:error, %Meta{errors: errors}} ->
    {:error, errors}
end

[woylie]

Can you try passing the for option to validate as well? Otherwise the validate function wouldn't know how to find the schema options. It doesn't return an error with Flop.validate (in contrast to MyFlop.validate) because the pagination type is not set to cursor.

case MyFlop.validate(flop, for: Entity) do

Yes, also, I think I can use MyFlop.filter and MyFlop.order_by to build a query and then pass it to ecto to perform the query without pagination like this

query
|> MyFlop.filter(flop)
|> MyFlop.order_by(flop)
|> MyRepo.all()

That should work.

I'm not entirely sure about the default limit handling. The reasoning for the new default limit and max limit was that Flop is meant to handle unsafe user-generated parameters, and it should apply a reasonable and safe default configuration. If a boundless query is indeed desired, that should be explicitly stated.

I was also thinking about a default_limit: :infinity option.

It is possible to pass pagination: false to the validation function, but that only prevents any pagination parameters from being cast. The default limit will still be applied. There are certainly arguments for and against this handling.

[lucacorti]

case MyFlop.validate(flop, for: Entity) do

Yes, this works fine against the main branch.

I'm not entirely sure about the default limit handling. The reasoning for the new default limit and max limit was that Flop is meant to handle unsafe user-generated parameters, and it should apply a reasonable and safe default configuration. If a boundless query is indeed desired, that should be explicitly stated.

I was also thinking about a default_limit: :infinity option.

I'm not sure about this, I'm fine with either a flag or calling Repo.all explicitly on the filtered query.