ListObjects terrible performance with `but not` exclusion about openfga

[KlausVii]

@jon-whit I have not had a chance to return to this for a while. I'll try and find some time in the near future.

[jon-whit]

@KlausVii how many objects is the ListObjects call for the project#can_view relation expected to return? If the number is high, then the higher latency is not too out of the ordinary. If an intersection a and b or an exclusion a but not b is involved in the relationship, we end up resolving the lookup with concurrent Checks. The more objects there are expected to be returned, then the worse the latency can become.

Also, for every Project that the user has the can_view relationship with, OpenFGA will have to resolve project:<id>#revoked relationship to determine the exclusion for project#can_view. If most of the projects being returned fall under the same tenant, then our upcoming work with Check caching (see #891) will help reduce those repetitive lookups. For example if you have tuples such as:

- tenant:acme#revoked:user:jon
- project:1#owner_tenant@tenant:acme
- project:2#owner_tenant@tenant:acme
- project:3#owner_tenant@tenant:acme
- ...

then when we go to resolve Check(tenant:acme#revoked@user:jon) in the project#revoked resolution path, then that lookup will be cached and re-used after the first resolution. This Check resolution cache will help a lot with your problem, and we expect to release that soon.

For us to better assist you though, could you share what your server settings are set to, what database are you using, what version of OpenFGA, etc..?

[KlausVii]

@jon-whit Thanks for the reply!

We expect on the order of 10-50 projects to be returned in most cases, from 1000s of projects in the tuples once we are fully up and running. Most users would probably not be revoked from anything usually.

I think the caching work will help substantially, since the main use case is that a user is revoked from a tenant and is therefore not allowed to access any of that tenant's projects.

Does the fact [user] is included under revoked in the project mean that fga would still need to check project:1#revoked@user:jon for every project anyway?

Is the current algo doing a separate revoked SQL query for each project in the list?

Another possible speed could be to internally list all projects a user can_view and then subtract all the revoked ones; these 2 queries are very fast individually without the but not in the model. Not sure how you would figure that out programatically though.

We are running:

• version 1.3
• default options
• postgres db
• inside an EKS cluster.

[jon-whit]

I think the caching work will help substantially, since the main use case is that a user is revoked from a tenant and is therefore not allowed to access any of that tenant's projects.

@KlausVii what is the expectation around "timeliness" of the revocation? For example, if you revoke a user from a project at the tenant or indiviual project level, how promptly should that revocation take effect for your application's use cases? Should it be as soon as it is revoked, or is there some buffer room for stale results? Do you have UI page renders that promptly and reactively display an unauthorized response if the users access changes? Do your UIs show lists of projects that the user can view and should those items be removed from the list promptly when a user has been revoked from the project? etc..

Does the fact [user] is included under revoked in the project mean that fga would still need to check project:1#revoked@user:jon for every project anyway?

Yes, unless we are able to quickly find another resolution path that determines user:jon is revoked at the project#owner_tenant level. If, for the most part, all of the projects have the same tenant ownership, then with query caching that alternative resolution path would resolve really quick and wouldn't require additional eval.

Another possible speed could be to internally list all projects a user can_view and then subtract all the revoked ones; these 2 queries are very fast individually without the but not in the model. Not sure how you would figure that out programatically though.

That is an option, and I don't want to discourage any creativity, but I wouldn't encourage you to do that 😄 . The reason for that is that the FGA model encodes the policy that should be enforced, and if you move the set logic outside of OpenFGA for that, then you effectively diminish the value of the policy as a the source of truth for the access that the system enforces. The goal should be to move policy logic outside of your apps into the model itself and let OpenFGA do the heavy lifting.

[KlausVii]

@jon-whit The revocation should be pretty quick, but having a lag of a minute or two shouldn't be too bad.

Our main demand for the revocation is to cascade down from the tenant, so I think we should be able to change the project revoked to just

    define revoked: revoked from owner_tenant

if that will improve perfomance come caching.

My last point was that maybe OpenFga could somehow figure out that intersection instead of doing all the individual checks, we want to rely on it as the sole source of truth.

For now we have removed all the but nots from the model and are instead removing all the user relations. Hope to be able to reinstate once the caching comes in.

[jon-whit]

The revocation should be pretty quick, but having a lag of a minute or two shouldn't be too bad.

In this case the caching approach in our upcoming release should suffice. You'll have staleness up to a configurable TTL, but it will significantly reduce the resolution of the tenant ownership lookups for all of the projects. If, in the future, your staleness requirements change to "immediately consistent" then we'll have to address that use case with future work. We've talked about introducing client-specified query consistency levels so that the client can choose what level of consistency the query should be served with.

My last point was that maybe OpenFga could somehow figure out that intersection instead of doing all the individual checks, we want to rely on it as the sole source of truth.

The problem with intersection and exclusion is that you, by definition, have to resolve both sets and then take the set intersection or set difference between them. You cannot short circuit that like you can for A union B. To evaluate A but not B you have to evaluate all of the objects of set A and set B and then compute their set difference.

What we do in OpenFGA for ListObjects, as a recent optimization we added in v1.2.0, is that we prune the evaluation of A and B to simply A and for each of the objects in set A we call Check on them. This is an optimization that is based on the set fact that size(A and B) <= min(size(A), size(B)). If set A is smaller than set B then it's cheaper to compute lookups on set A than it is set B, but if set A is larger then the computation is no worse than computing set A individually and calling Checks on the results to resolve the intersection with set B. Computing set A and set B and then explicitly taking their set intersection just adds more computation overhead and requires a memory footprint to keep those sets in memory before taking their intersection.

[jon-whit]

@KlausVii did the check query cache that we released in v1.3.1 help? If so, can we close? If not, I'd like to understand the behavior better so we can best assist. Are you still seeing the issue with v1.3.1 and the check query cache enabled?

[github-actions]

It appears this issue has been stale for at least 14 days 🗓️. If no action is taken the maintainer team may consider closing the issue. Please reach out if you need feedback or follow up actions from the maintainer team.

[miparnisari]

@KlausVii i'm going to close this issue, since it has become stale. If you are still experiencing problems, please open a new one. Thank you!