Comments (7)
Hi @deevashwer
Are you running a large model? There is a timeout config here, this can happen when data is kind of huge and send takes more than 100s.
It is possible that there are some network jitter that causes one of the node takes a little bit longer to recv data.
Thanks
from spu.
Yes, I'm running a large model in a LAN setting, so I don't expect significant jitters. It is a curious case because it works just fine for a few runs (say 4 or 5) and then on the 6th run, one of the sockets closes down. I'll try setting the timeout higher and let's see if that fixes the issue.
Thanks!
from spu.
That did not solve the problem. After a bunch of runs, the same error happened after around 1 hour and 43 minutes. One of the nodes gets automatically terminated with signal 9, and then the other two abort from a closed socket.
from spu.
Interesting, we'll try to reproduce this.
from spu.
Hi @deevashwer , we have encountered a some similar issue (it's in chinese) before due to a potential memory leak problem in glibc. Maybe you can have a try with a different version of glibc or tcmalloc.
from spu.
Hi @tpppppub. Thanks for the reference. Switching to tcmalloc unfortunately didn't resolve the issue. It does look like a memory leak however.
from spu.
@warriorpaw Can you take a look when you have time? Thanks
from spu.
Related Issues (20)
- How to use SPU to evaluate private models in 2PC setting with only one machine? HOT 16
- [Bug]: One more minus sign HOT 2
- [Bug]: Package 'examples/python/ml/my_custom_file' contains errors HOT 10
- [Bug]: The critical condition judgment is wrong HOT 5
- [Bug]: Error when trying to benchmark SPU latency in 2PC setting. HOT 3
- another case where secret indexing doesn't seem to work HOT 2
- [Bug]: 8x communication compared to reported in Cheetah HOT 5
- [Operation Question] How to separate truncation and matmul operations HOT 9
- [Question]: Are there any files building correspondence between the kernels and their dispatching functions? HOT 3
- [Bug]: bitintl_b in ab_api.cc is wrong HOT 3
- [Question]: The number of convolutional multiplication decreases but the communication cost increases in SPU HOT 4
- [Bug]: gcc 11.2下的编译问题 HOT 12
- [Question]: 能否不重复编译外部库,加速编译速度? HOT 4
- [Question]: stub_method方法实现将函数交由server执行,请问如何调试server中函数的具体执行过程? HOT 4
- [Question]: Common type of Ashare and Bshare HOT 1
- [Question]: Evaluating the model with significantly reduced number of ReLUs receives limited communication savings HOT 17
- [Question]: Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.0KiB (rounded to 3072) HOT 9
- [Usability]: 运行LLaMa时报错'Config' object has no attribute 'define_bool_state' HOT 10
- 我想请问一下bumblebee的复现配置 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spu.