Null value in column of relation table violates not-null constraint about efcore.pg

[mwpowellhtx]

Of note, when I take the value conversion away, the exception does seem to allude to the NRT suitability of the property mapping in question. Again... with the NRT, does not seem to consistently verify the actual value post-conversion, meaning the breakpoint does not land there.

[mwpowellhtx]

Further seems to be throwing from efcore BatchExecutor, as a function of the Npgsql being adapted in. Seems like an odd place to be encountering this issue, except perhaps as a function of pg integration.

[mwpowellhtx]

It is trying to do an INSERT first followed by an UPDATE? The infamouse pg upsert pattern, IIRC, from Marten experience, for example.

As a 🧠 experiment, I wondered if possibly the efcore mapping needed help in determining an appropriate upsert default default value. Which IMO renders native database behaviors such as DEFAULT "[]"::JSON completely and totally irrelevant. I mean what's the point of using those sort of clauses, if the ORM is going to want control over that? At any rate, with this bit here:

o.Property(p => p.Items).HasColumnName("itemsjson").HasColumnType("json").IsRequired().HasDefaultValue(Array.Empty<string>()).HasConversion(new ItemsValueConverter());
//                                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Without seemingly EVER consulting with the model ACTUAL VALUE not even once. IMO should be seeing the conversion happen in both directions but never see the ConvertTo except for the one initial time.

[mwpowellhtx]

This all being said, it seems to "work" when I help the mapping use cases. But it is a bit disconcerting that:

1. The pattern is not more model centric, consulting with actual property values
2. We cannot apparently make certain native database decisions, i.e. DEFAULT "[]"::JSON, even in the update-insert i.e. upsert scenario. What's the point of pg defaults if they would not actually take effect during an insert? Useless, completely worthless IMHO.

Your 🧠 ... Cheers 🍻

[mwpowellhtx]

Oh my here is the log, seems even worse than that. I do not know how either the Items or itemsjsoncolumn are being done here. They are apparently being completely ignored, pruned from the query through the ORM.

I never see breakpoints on either the property get or the value conversion ConvertTo step. Except the first time, observed in passing thinking "interesting" then ever again wondering "concerning", apparently falling back to NRT biased type validation.

2024-08-01 12:02:44.443 EDT [33688] LOG:  execute <unnamed>: INSERT INTO efcore_minpro_modeltemplate (id) VALUES ($1) RETURNING itemsjson
2024-08-01 12:02:44.443 EDT [33688] DETAIL:  parameters: $1 = '49585949-78b7-44c7-a7b7-7b9551557f14'

Anyone else seeing a problem with this in the log? I get the RETURNING part, which if I understand that, then does not see a reason to do the subsequent UPDATE, if that is truly pending. Maybe that happened in a completely other Task asynchronous worker thread, so my breakpoint in the main program context never lands, who knows 🤷‍♂️ .

[mwpowellhtx]

Out of curiosity, I add some trace logging, but those are not logging either, I would half expect them to.

public static string ConvertTo(ICollection<string> items)
{
    Trace.WriteLineIf(items is null, $"{nameof(items)} is null");
    Trace.WriteLineIf(items is not null, $"{nameof(items)} is not null");
    // Intentionally returning empty JSON array here, for sake of minpro example
    return "[]";
}

And in the property, along similar lines:

get
{
    Trace.WriteLineIf(_items is null, $"{nameof(_items)} value is null");
    Trace.WriteLineIf(_items is not null, $"{nameof(_items)} value is not null: Count = {_items.Count}");
    return _items ??= [];
    //            ^^^^^^
}

Neither of which is happening. I do see the property trace, when I debug inspect the model instance, but that is it.

Which tells me the ORM is never using that value. I'm not sure how that is even possible, unless perhaps a proxy class is being generated. Which from my NHibernate experience, for example, I know that can be thing, which commonly includes the active database code in the instance.

[mwpowellhtx]

Vis-a-vis the issue at hand and observations, wanting to know a few things, best I can estimate.

1. Are efcore and npgsql adapters making some assumptoins about, taking liberties regarding, underlying model, NRT-ness, and so on. It is our preference, obviously, for the operation to be as model centric as possible, with the efcore repository being a transparent as possible serialization layer.
2. Why is it that the mapping seems to require a default spec? Without which fails during (presumably) upsertion, throws an exception, presumably around the NRT question, but that is not clear, and certainly not based on the actual values in play.
3. Why is it the mapping and DbSet<T> are entirely ignoring, it would seem, bypassing the model property getter; unless, as speculated here, that is happening during an async or Task oriented worker thread. The truly perplexing thing is, we get no Trace responses.

The implications of which, just trying to understand the whats, hows and why fors, if you will, because that will determine the modes in which we are expected to operate, versus efcore, etc, in order to do the work we want it to do.

For @roji or really anyone with insights around the question.

Will have my minpro updated shortly with the current revelations, observations, introspections, etc.

Thanks for bearing with me, it has been rather educational the idiosyncacies of both efcore as well as npgsql and pg in general.

[mwpowellhtx]

Summarizing above, besides the NRT default value versus NOT NULL constraint concerns... The main objection is during the DbSet<T>.Add, is completely one hundred percent disrespecting the CURRENT OBJECT STATE. It seems to be reaching down to the database layer for its default values whatever those happen to be, and using those. With complete disregard for the model instance we gave it in the first place, whatever however those property values shall have been furnished, calculated, internally defaulted, and so on. The only way we have found to ensure that is not the case, is to walk each of the properties, and ensure we do the corresponding get. I'm not sure, is there some npgsql or otherwise efcore policy which may be set which does the right thing in this regards? Again, kindly pardoning my ignorance, maybe there are good reasons for why it is doing what it is doing, I do not know. Perhaps educate me in this area.

[mwpowellhtx]

I think I see what might be happening, but I think the idea of field preference is dangerous.

modelBuilder.UsePropertyAccessMode(PropertyAccessMode.Property);

[mwpowellhtx]

I'm satisfied I think the property field confusion was the major solution in this area.

Still the NRT (or lack there of), default values portion, could use some clarification. My guess is, the type validation taking liberties with non-NRT produced IL and so forth. Not sure why there also needs ot be a default value specific, when the model furnishes its properties, but whatever.

Cheers 🍻

[roji]

@mwpowellhtx I've quickly read through this issue and through npgsql/npgsql#5797 - there's a lot going on and I don't quite understand what exactly you're asking, and if you figured it out or have further issues. If you do, can you please post a minimal repro and describe one precise problem you have at a time?