redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer. about diskover-community

[shirosaidev]

Try with rc3 and let me know if you still get this error.

May i know the reason for above error. That will helps me in future.

[shirosaidev]

did you try rc3? no more error?

[shirosaidev]

rc3 has fixes to hopefully make this error go away, before rc3 directory batch size could get too large using -a so I set maxsize of batch size to limit data added to Redis.

Thank you. Let me try with rc3 and let you know the status.

With rc3 scan was completed successfully. Thank you.

[abhimanyu3-zz]

rc3 ??? you mean redis server...I am having that but still getting the issue

Redis server v=3.2.0 sha=00000000:0 malloc=jemalloc-4.0.3 bits=64

[shirosaidev]

What version of diskover are you running? Are you cacheing dir times in redis (diskover.cfg setting)?

[shirosaidev]

Try updating to redis 4.0.9 and crawl with diskover 1.5.0-rc17.

i'm rebuilding complete thing from scratch. give me some time. i will re-initiate with new rc17 and share you the result.

Do we have any option to perform one time incremental scan once a day without deleting old indexes. instead of continues scan.

[shirosaidev]

there is currently no incremental indexing, usually people just create a new index daily or weekly using a different name and then can compare data between two indices to see change using diskover-web.

[shirosaidev]

please try latest rc21 and see if you get these issues with redis, there is a new setting in diskover.cfg.sample for redis "ttl" (time to live) which you could try setting to lower value than default 500 to store keys/results in redis. You could also set maxmemory and memory policy in redis conf to evict keys which are no longer needed by diskover. Also, check if cachedirtimes is set to False in your diskover.cfg redis section to reduce redis memory usage.