Null error in QueryArchetypeEnumerator about arch

[stgeorge]

I said I was going to take a break but I had a few more minutes, lol

So the issue is basically while inside a foreach of a query.GetArchetypeIterator(), the PooledList entries it is referencing could get moved, so any call to GetOrCreate could pull the rug out from under...

It looks like you might not be able to use "ref var archetype" due to that small risk the underlying loop could get moved (changing it "var archetype" does seem to fix it). That seems like a high price to pay...

Alternatively, it looks like raising the Archetypes PooledList initial capacity from 8 to something higher (in my case 32) would reduce the chances of this drastically. Maybe these limits can be configurable by the end-user when creating their world - not sure the memory impacts of this:

    Archetypes = new PooledList<Archetype>(32);

[stgeorge]

I will try it out and no worries, appreciate all your efforts and good luck on your thesis!

[stgeorge]

As for your other suggestions, I initially started with the null check, but it didn't feel quite right - I'll leave that up to you to decide.

PooledList is valuable to avoid allocations during a game loop but since this use case will allocate anyway if it exceeds capacity then it does kind of defeat the purpose a bit if archetypes aren't meant to be deleted to re-use the pool.

[genaray]

I took a quick look at it and set the clearmode to never. This should hopefully prevent it for now. I also made sure that the archetypes nevertheless are disposed correctly and freed so no memory leak should occur :)

I couldn't change it to a normal list since there problems with spans for .NetStandard2.1, so i kept it a pooledlist for now.

[stgeorge]

Initial tests look positive, jacked up the entities to 1000 per wave and working well so far, player dies without issue - killing loads of entities, got up to 3000 entities or so and frame rate was still over refresh rate (whole reason I'm using this lib!)

Looks good so far! Feels pretty stable, will keep on testing it out as I add more stuff. Thanks!

[genaray]

Thanks? Im gonna investigate this... However this is very weird, since archetypes are never removed or deleted which prevents them from becoming null :o

Do you trim the world to release resources?

[stgeorge]

Not using TrimExcess. From what I can tell, adding the new component creates a newArchetype and leaves the oldArchetype alone.

Add(in Entity entity, in T cmp)
GetOrCreate(oldArchetype.Types.Add(typeof(T)))
Move(in entity, oldArchetype, newArchetype, out var slot);

then it goes back to the query enumerator, and eventually it tries to get the Current archetype (which is a span pointing into a PooledList). But for some reason the _ptr for every possible index of _length is null. Is it possible that PooledList got cleared out for some reason and recreated using a different area of the PooledList??

I removed my other fix and changed this line:

Archetypes = new PooledList<Archetype>(8);

to

Archetypes = new PooledList<Archetype>(8, ClearMode.Never);

And amazingly the bug seems to go away... looking up that setting the default behavior is ClearMode.Auto, which in .NET Core is the same as ClearMode.Always. So, is it possible GC is doing something here? I'm not sure how viable this is as a fix though as I don't have enough understanding of what's happening internally. Hoping it helps.

  public enum ClearMode
  {
    /// <summary>
    /// <para><see cref="F:Collections.Pooled.ClearMode.Auto" /> has different behavior depending on the host project's target framework.</para>
    /// <para>.NET Standard 2.1, .NET Core 3: Reference types and value types that contain reference types are cleared
    /// when the internal arrays are returned to the pool. Value types that do not contain reference
    /// types are not cleared when returned to the pool.</para>
    /// <para>.NET Standard 2.0, .NET Framework: All user types are cleared before returning to the pool, in case they
    /// contain reference types.
    /// For .NET Standard, Auto and Always have the same behavior.</para>
    /// </summary>
    Auto,
    /// <summary>
    /// <para>The <see cref="F:Collections.Pooled.ClearMode.Always" /> option has the effect of always clearing user types before returning to the pool.
    /// This is the default behavior on .NET Standard 2.0 and .NET Framework.</para><para>You might want to turn this on in a .NET Core project
    /// if you were concerned about sensitive data stored in value types leaking to other pars of your application.</para>
    /// </summary>
    Always,
    /// <summary>
    /// <para><see cref="F:Collections.Pooled.ClearMode.Never" /> will cause pooled collections to never clear user types before returning them to the pool.</para>
    /// <para>You might want to use this setting in a .NET Core 3 or .NET Standard 2.1 project when you know that a particular collection stores
    /// only value types and you want the performance benefit of not taking time to reset array items to their default value.</para>
    /// <para>Be careful with this setting: if used for a collection that contains reference types, or value types that contain
    /// reference types, this setting could cause memory issues by making the garbage collector unable to clean up instances
    /// that are still being referenced by arrays sitting in the ArrayPool.</para>
    /// </summary>
    Never,
  }

[stgeorge]

Reading more, it seems like it might not be a good idea to use ClearMode.Never with a reference type.

[stgeorge]

So after some trial and error, I narrowed it down to this line in GetOrCreate:

    Archetypes.Add(archetype);

before this line is run, the ref to the old archetype is populated and after this line it is null.

Digging deeper, the Add is part of PooledList, and looks like this:

  ++this._version;
  T[] items = this._items;
  int size = this._size;
  if ((uint) size < (uint) items.Length)
  {
    this._size = size + 1;
    items[size] = item;
  }
  else
    this.AddWithResize(item);

the bug only seems to happen when the AddWithResize is run, which means it is probably moving the data to a different area of the pool. Later on, the code to actual return the updated array clears the old part of the pool (based on the ClearMode setting).

So... I'm not yet sure what the real fix is here yet, I'll take a break for now but hopefully this provides some more insight.

Also, just want to make sure you know my project is just for fun and I'm not expecting anything - just want to help improve the library and hope to see it gain traction - there's no expectations on my part :) Thanks for making this and enjoy your weekend!

[genaray]

I said I was going to take a break but I had a few more minutes, lol

So the issue is basically while inside a foreach of a query.GetArchetypeIterator(), the PooledList entries it is referencing could get moved, so any call to GetOrCreate could pull the rug out from under...

It looks like you might not be able to use "ref var archetype" due to that small risk the underlying loop could get moved (changing it "var archetype" does seem to fix it). That seems like a high price to pay...

Alternatively, it looks like raising the Archetypes PooledList initial capacity from 8 to something higher (in my case 32) would reduce the chances of this drastically. Maybe these limits can be configurable by the end-user when creating their world - not sure the memory impacts of this:

    Archetypes = new PooledList<Archetype>(32);

Thanks a lot! :) Did not that have that much time yet to look into it (I'm currently preparing my bachelor thesis so I'm quite busy).

However a simple change from ref var archetype to var archetype shouldn't be that expensive. Both are still reference types and thus its performance will stay the same. I'll look into this quickly and see how it performs :)

[genaray]

Changed some of the underlaying enumerators and automatic generated query to make use of var archetype :)
Would be great if you could try it out to see if the issue persists ^^

[stgeorge]

Sadly, while it seems to happen less, it still does happen under stress. It seems like a reference into the pooled list could still get moved by some means making the spans into it no longer valid.

The only thing that seems to work consistently is to ensure enough capacity so archetype pooled list never needs to resize. Since it sounds like old Archetypes don't need to get deleted anyway, maybe raising the initial capacity to something unreachable for most projects like 64 (or letting initial capacities be set during World Create) would be more resilient - haven't really looked at what that means in terms of total memory use.

For whatever reason it seems going from 8 to 16 archetypes doesn't trigger it as much, but going from 16 to 32 does, so I've been using 32.

[genaray]

Hmm thats weird. Im gonna look further into this. Sadly its kinda hard to reproduce, right? Probably we could just insert a null check for each archetype, or remove that pooled list. Probably it doesn't even make sense to pool that list.

[stgeorge]

I think I got lucky when I made that change from ref var to var and it happened to work. But then I made the change to use the higher capacity and never ran into it again.

So for me, it happens when I have 16 archetypes and add another component to an entity that requires a new archetype to be made and it resizes from 16 to 32.

I used Resharper disassembly to be able to put a breakpoint in the PooledList implementation of Add on the line:

this.AddWithResize(item);

so whenever it hits that, then I think things go sour sometime after that (usually pretty soon afterwards if you're already inside of an enumerator using that same PooledList).

[genaray]

Initial tests look positive, jacked up the entities to 1000 per wave and working well so far, player dies without issue - killing loads of entities, got up to 3000 entities or so and frame rate was still over refresh rate (whole reason I'm using this lib!)

Looks good so far! Feels pretty stable, will keep on testing it out as I add more stuff. Thanks!

Glad to hear that! :) Lemme know if there more iusses!