Segmentation fault with `Potential()` about pytreegrav

[mikegrudic]

Pretty puzzling, can't say I've seen that one. If you could dump the exact arrays provided to Potential() that cause it to segfault I could possibly check if it's a reproducible algorithmic issue.

[ktyaman]

Here are the snapshot, python script and my C library (before and after compiled).
I'm sorry but the snapshot file has a size of 952 MB.

I added your gmail to the list who can access these files.
https://drive.google.com/drive/folders/1eWbnFLpCgIQ6nRcfz_Jeo4VSbHtH5nT4?usp=sharing

(I rewrote the script so that you can immediately run it. I have checked that it runs into segfault.)

[saluto]

I haven't checked whether the above segmentation fault is caused by the same bug as mine, but here is my situation:

After disabling Numba's JIT, I found that an index error is raised here due to requiring more than 2*self.NumParticles nodes (i.e. new_node_idx >= len(self.Coordinates)), as were initialized here. A small example:

import numpy as np
import pytreegrav as tg

pos = np.array([
  [0., 0., 0.],
  [1., 0., 0.],
  [10., 0., 0.],
])
m = np.ones(len(pos))
# This crashes.
pot = tg.Potential(pos, m, method="tree")

In my case I fixed it by doubling the number of nodes (copying all arrays over to new arrays) before we would otherwise run into the index error. But there might be a more efficient solution.

[mikegrudic]

Hi guys,

I believe I just fixed the above bug from @saluto in the latest push. The issue is that for an octree structure you actually cannot put an a priori upper bound on the number of nodes, because two points that are close enough together can demand more recursive node splitting than average when building the tree (e.g. 1 and 0 above). But now, pytreegrav will dynamically allocate more storage in the tree structure if required. This has the additional benefit of allowing us to run a bit leaner on memory in the average case.

@ktyaman I was able to reproduce your error. The problem was that your dataset - if I parsed it correctly - consists of 8075000 points, but only 2 unique ones. pytreegrav can handle limited repetitions of points but if the entire dataset is like this then it ends up building incredibly deep trees and running out of memory. So the actual crash was fundamentally the same as @saluto's, but to fix it for this specific application would require some reworking.

I really appreciate both your help.

Mike