alexyorke / spider Goto Github PK

There is a memory leak somewhere, and it cases a ~1.25mb/hr additional memory usage. It is possible that the close() handlers are not being called by the garbage collector, or the crawlers aren't being disconnected from the delegates.

If Spider is left to run over a period of a day or more, it will generate hundreds of empty folders, and eventually stall, not joining any more worlds or downloading any more events.

Workaround: make a cron script to delete empty folders everyday.

Right now, Spider is a monolithic architecture and makes it difficult to test the outcomes of how specific crawlers behaviours' are changed. Converting to a process-per-crawler would allow each crawler to be individually changed and tested, and the client would not have to be restarted.

It would also allow levels to be crawled on demand.

Inside of the .sln file, the configuration is set to load the .csproj from "EventDebugEE", it should be changed to "Spider".

This can be easily done with just notepad.

Make the crawler's OnMessage delegate a wrapper for the one that writes the event to the queue, so that the timings and physics will be more accurate. Instead of the events queuing up without timestamps, they will queue up with time stamps, and can be processed at virtually any time.