Proposal: Change negative filters to include null results about graphql

[darrellwarde]

Hi @wheresrhys, apologies for the long-term radio silence on this one. Addressing how both @neo4j-graphql and neo4j-graphql-js treat null values, I personally believe that they are handled as they should be at the moment. Due to them both essentially being GraphQL to Cypher translation engines, null values are treated as they are in Cypher, which is essentially that null equals non-existence (as you are already aware from reading the above messages).

However, I don't deny that there is definitely going to be use cases for treating null values differently, but this is very much a business logic decision, and it sounds like in your business, lifecycleStage not being set has a particular meaning that you need to be expressed through your API.

I have some initial thoughts on this, which is that we could introduce some new field directives which could be used to help in these situations:

• @default(value: "DEFAULT VALUE") - when creating a node, if any fields have this directive, set the value of the property in the node to this value
• @coalesce(value: "DEFAULT VALUE") - when querying a node, wrap any properties with a COALESCE statement in the WHERE clause, if the field is decorated with this directive

I believe that this would help address the issues you were having. I'm going to write up a ticket (I don't think the work will be too complicated), and I'll see about getting this done for you.

[darrellwarde]

I should maybe add, that for 90% of cases, if you need the non-existence of a property to have intent, then that field should be mandatory in your GraphQL type definitions and always have a value.

However, I do realise that data is complicated, especially with legacy datasets, so I can see the value in having something like the above available.

[wheresrhys]

Would it make sense to have a single @default directive with parameters to express when to apply the default (write, filter, ...), and it would apply whichever implementation(s) the parameters turn on.

[danstarns]

Hello. Can you give an example of how you would like to do the above? is it that you only want to provide lifecycleStage_not: Decommissioned ?

[wheresrhys]

Yes that's right.

If my data is

• System {id: 1, lifecycleStage: Production}
• System {id: 2, lifecycleStage: Decommissioned}
• System {id: 3}

I would like

	Systems(filter: {
		lifecycleStage_not: Decommissioned
	}) {
		id
	}

to return {data: Systems:[{id:1}, {id:3}]}, but neo4j-graphql-js returns {data: Systems:[{id:1}]}

[wheresrhys]

There is a similar issue with _some and _every filters

e.g. if my System has no dependencies then it will not be retrieved by this query:

{
  Systems (filter:{
    dependencies_every: {
      id_not:"asdasdas"
    }
  }) {
    	id
  }
}

I'm not sure what I think the right thing to return here is, but I think it's reasonable to suggest the every rule running over an empty array should be truthy (in JS [].every() behaves like this).

I think the alpha stage of this new project is an ideal point at which to put some proper thought into how filters should behave when dealing with null values; what would be the least surprising and most useful behaviour?

[wheresrhys]

Thanks for the response. I'll get back to you once I've had time to consider if your proposals would help.

[wheresrhys]

Having taken a look at the generated queries I see your point about ti being a minimally opinionated graphql -> cypher translation, and I guess this is the behaviour that falls out of that. I really wish it weren't so, but I guess you can't please all the people all the time.

if you need the non-existence of a property to have intent, then that field should be mandatory in your GraphQL type definitions and always have a value

The only intent it has is 'nobody entered a value'. I'm not sure I agree with your statement as it feels tantamount to reimplementing null using a custom value. I also think it would be difficult to apply that principle to relationships. Do you have any references to using null to represent non-existence being an anti patten in GraphQL (genuinely interested to understand the thinking behind it).

I'm not sure the @default and @coalesce ideas would work for us, although I don't fully understand the @coalesce suggestion. Maybe I'll DM you in slack to discuss further

Another idea for a solution - user defined filter patterns passed in as an option to the neo4jGraphQL constructor. e.g.

new Neo4jGraphQL({
	filterPatterns: {
		not_or_empty: '(NOT `system`.{fieldName}=  $filter.{fieldName}_{filterPatternName}) OR (NOT EXISTS(`system`.{fieldName}) )'
	}
});

... where the template-like bits will have values inserted when enhancing the schema/executing resolvers

Not sure how viable that is, but just throwing it out there

[wheresrhys]

Or in fact, how open would you be to adding a not_or_empty filter as standard?

[darrellwarde]

Excuse any delays on my end also, this really is a problem that takes a lot of consideration and doesn't appear to have a perfect solution!

The only intent it has is 'nobody entered a value'. I'm not sure I agree with your statement as it feels tantamount to reimplementing null using a custom value. I also think it would be difficult to apply that principle to relationships. Do you have any references to using null to represent non-existence being an anti patten in GraphQL (genuinely interested to understand the thinking behind it).

I don't have any examples, and I don't really believe it to be an anti-pattern. This is more about, as you say, making sure that our GraphQL to Cypher translation isn't too opinionated.

I'm not sure the @default and @coalesce ideas would work for us, although I don't fully understand the @coalesce suggestion. Maybe I'll DM you in slack to discuss further

Feel free to DM me in Slack, happy to have a call about this as well if it would benefit you. My belief is that @coalesce would allow you to use non-existent properties in comparisons, without explicitly setting those properties like using @default would. For your example, the Cypher WHERE clause would end up something like WHERE (NOT COALESCE(this.lifecycleStage, "Undefined") = "Decommissioned"). "Undefined" could be anything here, it just means that it's going to be included in the comparison. This also means that once you have specified @coalesce on the field, you can just query it as you want to with no worries, and it won't change the underlying data.

Another idea for a solution - user defined filter patterns passed in as an option to the neo4jGraphQL constructor. e.g.

new Neo4jGraphQL({
	filterPatterns: {
		not_or_empty: '(NOT `system`.{fieldName}=  $filter.{fieldName}_{filterPatternName}) OR (NOT EXISTS(`system`.{fieldName}) )'
	}
});

... where the template-like bits will have values inserted when enhancing the schema/executing resolvers

Not sure how viable that is, but just throwing it out there

I do like this idea! Definitely not something we can do right now, I imagine this would get quite complex quite quickly due to the recursive nature of how we generate Cypher. But I will note it down, there's definitely some value in having "pre-canned" queries if you know you're going to be using them all over the place.

Or in fact, how open would you be to adding a not_or_empty filter as standard?

Personally, I'm not a fan, but maybe I'm just not a fan of the naming convention. Lots of food for thought here!

@danstarns, would you care to chime in on any of this?

[wheresrhys]

Just tried out the coalesce approach in neo4j browser and it does return the result set I need 🎉

To clarify, you're proposing that this coalesce directive only take effect in filters, or would it have some affect on properties returned?

Do you envisage other scenarios where having it as a directive would be useful? If you implement the directive it would solve my problem and I would definitely use it, but if the directive is only implemented in order to solve my problem I'd prefer (and it's probably similar complexity to implement) an option to enable globally - nullSafeFilters: true or something - rather than have to add locally to most/all properties in my schema.

[darrellwarde]

Just tried out the coalesce approach in neo4j browser and it does return the result set I need 🎉

Great news! 🎉

To clarify, you're proposing that this coalesce directive only take effect in filters, or would it have some affect on properties returned?

Indeed, only in filters - @default would be the one to use if users wanted something different to be returned.

Do you envisage other scenarios where having it as a directive would be useful? If you implement the directive it would solve my problem and I would definitely use it, but if the directive is only implemented in order to solve my problem I'd prefer (and it's probably similar complexity to implement) an option to enable globally - nullSafeFilters: true or something - rather than have to add locally to most/all properties in my schema.

At the moment, this is the only kind of scenario where I can imagine this would be useful, but I reckon you're far from the only person who would be after this functionality.

Whilst I can see the value in a global setting for this, the output Cypher would be a trainwreck. In my opinion, this @coalesce solution is concise and effective, it's only applied where it needs to be, and it's more consistent in terms of how it is applied in the Cypher. I wouldn't feel comfortable using COALESCE in Cypher without users providing the default value which they would like to use, so a global setting would mean we have to use other methods which are much more complicated and not as consistent (_NOT would be totally different to _NOT_IN for instance).

[wheresrhys]

Well I'm happy then. Our schema is autogenerated from some config so it's a write once, run many times task for us anyway.

Thanks for listening 😄

PS we can't try out the library in earnest until relationship properties are implemented anyway, so don't rush to implement these on my behalf

[wheresrhys]

One final thought - is @coalesce a name too guided by the implementation, rather than the purpose? Not sure what else I'd call it though.

[darrellwarde]

Well I'm happy then. Our schema is autogenerated from some config so it's a write once, run many times task for us anyway.

Thanks for listening 😄

Of course, no need to thank me!

PS we can't try out the library in earnest until relationship properties are implemented anyway, so don't rush to implement these on my behalf

Damn, I'm sad to hear that, I'm gutted we're not going to be able to get it in for the 1.0.0 release. I know you already have, but please do comment on neo4j/graphql-tracker-temp#7 and keep up-to-date with the rumblings in there to make sure you're happy with the direction we're taking.

One final thought - is @coalesce a name too guided by the implementation, rather than the purpose? Not sure what else I'd call it though.

I completely agree, it's an absolute leak of the underlying implementation! But likewise, I can't for the life of me think of what else I might call it given that @default is going to have a very different meaning!

[darrellwarde]

This has been considered, however, the proposed @default and @coalesce do completely different things despite having similar outcomes. @default will simply add default values to GraphQL input types, whilst @coalesce affects the translation to Cypher.

Additionally, in my opinion, directive arguments should work in tandem with one another. If we were to do something like this, then filter would become completely redundant if write is specified, which to me means they should have never been made into one directive in the first place.

But I think it's all a matter of personal preference really, I'm still mulling this one over!

[darrellwarde]

@default and @coalesce are now available in version 1.0.0-alpha.6, so I'm going to close this issue now. Please give them a try to see if they meet your requirements, and raise another issue if we need to refine further!

[wheresrhys]

Thanks for addressing it 😄 .

As mentioned in other issues around relationship properties, I'm unable to fully adopt the library at the moment, but I will try to spike using it for a subset of our current features over the next couple of months in order to feedback on this implementation.