Growing memory usage about h3-js

[nrabinowitz]

Do you have a reproducible case we can test with here? We did have an earlier memory issue we believe to be fixed, around polyfill. On the Emscripten size, there's a fixed memory limit, so I believe running a bad case repeatedly should eventually error out with Emscripten's version of OOM - so all we need is an example of something bad that we can run millions of times.

[johanschram]

Ok, here is an exemple of an isochrone that rises the issue.
Of course, there is always the possibility that I don't use this library as I should...

const isochron = require('./isochrones.json');
for (let i = 0 ; i < 5000; i++){
    isochron.features[0].geometry.coordinates.forEach((singlePolygon) => {
        let xxx = h3.polyfill(singlePolygon, 9, false)
        xxx = null;
    })
}

isochrones.zip

Worth mentioning:
node: 12.16.1
h3-js: 3.7.1

[nrabinowitz]

OK, I can take a look. Does this give you an OOM error? or do you just see the memory rising in system monitors?

[johanschram]

Both, memory is rising and depending on how much memory is given to node, eventually you will get an OOM error.

[nrabinowitz]

OK, but currently this is a Node OOM, not an Emscripten OOM. That's interesting, Emscripten has a dedicated amount of memory for its internal "heap" and should throw its own OOM error well before Node runs out of memory. That suggests this might be on the JS side. I can try to do some tests today.

[nrabinowitz]

I am not able to repro this. I ran the code above and charted the memory output using process.memoryUsage(). I see the expected sawtooth - memory ramping up, then getting garbage collected and going down. The bottom of the graph seems to stay pretty stable at 4-5Mb.

[4999] memoryUsage: 17777 / 38568 kB | max 20590 kB

   20590.00 ┤  ╭╮   ╭╮           ╭╮   ╭╮           ╭╮   ╭╮           ╭╮   ╭╮           ╭╮   ╭╮           ╭╮   ╭╮           ╭╮   ╭╮   
   17989.67 ┤  ││ ╭╮││ ╭╮   ╭╮   ││ ╭╮││ ╭╮   ╭╮   ││ ╭╮││ ╭╮   ╭╮   ││ ╭╮││ ╭╮   ╭╮   ││ ╭╮││ ╭╮   ╭╮   ││ ╭╮││ ╭╮   ╭╮   ││ ╭╮││ ╭ 
   15389.33 ┤╭╮││ ││││ ││ ╭╮││ ╭╮││ ││││ ││ ╭╮││ ╭╮││ ││││ ││ ╭╮││ ╭╮││ ││││ ││ ╭╮││ ╭╮││ ││││ ││ ╭╮││ ╭╮││ ││││ ││ ╭╮││ ╭╮││ ││││ │ 
   12789.00 ┤│││╰╮│││╰╮││ │││╰╮│││╰╮│││╰╮││ │││╰╮│││╰╮│││╰╮││ │││╰╮│││╰╮│││╰╮││ │││╰╮│││╰╮│││╰╮││ │││╰╮│││╰╮│││╰╮││ │││╰╮│││╰╮│││╰╮│ 
   10188.67 ┤│╰╯ ││╰╯ ││╰╮│╰╯ ││╰╯ ││╰╯ ││╰╮│╰╯ ││╰╯ ││╰╯ ││╰╮│╰╯ ││╰╯ ││╰╯ ││╰╮│╰╯ ││╰╯ ││╰╯ ││╰╮│╰╯ ││╰╯ ││╰╯ ││╰╮│╰╯ ││╰╯ ││╰╯ ││ 
    7588.33 ┤│   ╰╯   ╰╯ ││   ││   ╰╯   ╰╯ ││   ││   ╰╯   ╰╯ ││   ││   ╰╯   ╰╯ ││   ││   ╰╯   ╰╯ ││   ││   ╰╯   ╰╯ ││   ││   ╰╯   ╰╯ 
    4988.00 ┼╯           ╰╯   ╰╯           ╰╯   ╰╯           ╰╯   ╰╯           ╰╯   ╰╯           ╰╯   ╰╯           ╰╯   ╰╯           

This shows high memory usage, but I don't see a leak, and I went over the code and don't see leak suspects (though this is a particularly hairy part of the code w/r/t memory management). I used Node v14.16.1.

What Node version are you using? And does the isolated example above throw an OOM for you? We're definitely using a lot of memory here, and you could likely reduce it by e.g. splitting up the polygons or using a coarser hex and then uncompacting it to a finer resolution, but I'm not sure I see a leak.

[nrabinowitz]

Closing as I was never able to repro. Please reopen if there's a repeatable scenario.