[Bug]: gRPC Socket Shutting Down After Many Runs about spu

[anakinxc]

Hi @deevashwer

Are you running a large model? There is a timeout config here, this can happen when data is kind of huge and send takes more than 100s.

It is possible that there are some network jitter that causes one of the node takes a little bit longer to recv data.

Thanks

[deevashwer]

Yes, I'm running a large model in a LAN setting, so I don't expect significant jitters. It is a curious case because it works just fine for a few runs (say 4 or 5) and then on the 6th run, one of the sockets closes down. I'll try setting the timeout higher and let's see if that fixes the issue.

Thanks!

[deevashwer]

That did not solve the problem. After a bunch of runs, the same error happened after around 1 hour and 43 minutes. One of the nodes gets automatically terminated with signal 9, and then the other two abort from a closed socket.

[anakinxc]

Interesting, we'll try to reproduce this.

[tpppppub]

Hi @deevashwer , we have encountered a some similar issue (it's in chinese) before due to a potential memory leak problem in glibc. Maybe you can have a try with a different version of glibc or tcmalloc.

[deevashwer]

Hi @tpppppub. Thanks for the reference. Switching to tcmalloc unfortunately didn't resolve the issue. It does look like a memory leak however.

[anakinxc]

@warriorpaw Can you take a look when you have time? Thanks