Investigate the memory issues in production about krihelinator

[Nagasaki45]

d95beb7 didn't solve the issue.

[Nagasaki45]

What I understand so far

The issue is with large binaries. Binaries larger than 64 bytes are stored by erlang on a shared heap instead of on the process heap. GC should run on all of the processes that hold a reference to a large binary before it's memory can be reclaimed. See Erlang in Anger for more information about this.

I can notice the problem using the erlang observer on my local machine. In the Load Charts tab, the Memory Usage pannel show a steady growth of Binary data during Krihelinator.Periodic.scrape_from_bigquery and Krihelinator.Periodic.rescrae_still_dirty. In fact, it's not a memory leak, because every now and then (~few minutes) the binaries memory is freed.

In contrary with Erlang in Anger suggestion, it doesn't seem to be an issue with multiple processes holding the data, because calling :erlang.garbage_collect(pid) on the process that runs the Krihelinator.Periodic.run function free the memory. A possible solution (but ugly hack) is to trigger GC periodically as suggeste here.

[Nagasaki45]

591ce1a implements the periodic GC hack.
I asked about the issue on SO.