Terrible memory usage peak in magneticod about magnetico

[boramalper]

#56 fixes #55 but not this memory issue, am I correct? Also, did this happen before or after (or both times) after you merged #56?

As you said, this is clearly unacceptable but I don't know what might be causing it. I might guess that, since it occurs within 300-400 seconds after each time, it is because magneticod caches all info hashes in the database to prevent downloading them again. Although each info hash is 20 bytes, and even though you had 5,000,000 torrents, that would be around 95 MiB,† ans that might not be the real cause either. Do you have any guesses?

I can create detailed & structured (debug) logging messages in the following version, which can be enabled by a command-line option, to try to find the cause.

† This is also too much, and I'm planning to use a high-performance bloom filter implementation instead in the future.

[boramalper]

Found it after writing the above comment: =)

https://github.com/boramalper/magnetico/blob/v0.3.0/magneticod/magneticod/bittorrent.py#L219

It's probably this line. A malicious or simply malfunctioning peer might send a huge metadata size, which we are blindly accepting now (and allocating immediately).

I'll change it to accept only under a certain, reasonable size.

[ad-m]

@boramalper , feel that this error is not related to the earlier. For previous bugs the application did not work long enough for me to debug memory problems.

I use magneticod version as in pull requests #56 (git fetch origin pull/56/head:descriptor && git checkout descriptor). I am unable test that without #56 due short living process (exception was raised without that).

[ad-m]

@boramalper , do you able to have long running magneticod or do you continuous restart them as documented?

[boramalper]

@ad-m I continuously restart it.

[dessalines]

I'm getting this issue now too. Is there no way to give magneticod memory restrictions? A DigitalOcean box with 1GB of memory is killing it.

[ad-m]

@dessalines , do you use repo version or v0.3.0 (latest releases)? Magnetico* v.0.3.0 doesn't contains fix for that. @boramalper , do you can make a new release?