Comments (7)
Are you sure you add cores rather than threads? What is the size of the graph you are trying to process? Most likely it's a memory issue.
from deepwalk-c.
I increase threads parameter while running on 64 core machine.
from deepwalk-c.
If the size of the graph is really large, there might still be memory issues, especially in NUMA architectures. Can you (roughly) specify the size of the graph, and memory system used?
from deepwalk-c.
Sorry for not starting with numbers first.
./deepwalk -input in.bcsr -output out.model -threads 64 -dim 300 -nwalks 1000 -walklen 5 -window 3 -seed 4242 -verbose 2
nv: 747556, ne: 861549483
PR estimate complete
Using vectorized operations
Constructing HSM tree...
Done! Average code size: 20.7494
lr 0.000002, Progress 100.00%
Calculations took 8225.28 s to run
./deepwalk -input in.bcsr -output out.model -threads 8 -dim 300 -nwalks 1000 -walklen 5 -window 3 -seed 4242 -verbose 2
nv: 747556, ne: 861549483
PR estimate complete
Using vectorized operations
Constructing HSM tree...
Done! Average code size: 20.7604
lr 0.000002, Progress 100.00%
Calculations took 12681.46 s to run
I ran it on 64 cores & 256Gb mem ec2 instance. The graph bcsr file is 3.3Gb. The process consumes 5.5Gb of memory while runnnig.
from deepwalk-c.
From this limited information my most probably cause is memory access time. There is single DRAM contoller fetching graph (and embedding) parts from the memory, causing a bottleneck. This is probably not solvable unless you change the algorithm. If you are okay with higher memory consumption, you can consider an implementation that caches random walks in the memory.
On completely different issue, parameters you use seem quite a bit off. Why are you chaning the defaults?
from deepwalk-c.
You mean -nwalks 1000 -walklen 5 -window 3
? I just playing with parameters to see how it behaves.
from deepwalk-c.
If you want to see where the problem comes from, I would recommend running the process under GNU perf (tutorial here http://www.brendangregg.com/perf.html). I would expect that the running time is dominated with the memory access.
from deepwalk-c.
Related Issues (11)
- Multiple memory leaks and memory corruption HOT 3
- Source code decoupling HOT 3
- The performance problem HOT 1
- Node (names | ids) in the output of the deepwalk-c? HOT 1
- when use GPU,can speed up this? HOT 1
- how to evaluate of the correctness of the results? HOT 3
- How to read the output file by Python HOT 2
- Segmentation fault HOT 6
- Where is the convert-bcsr tool? HOT 1
- Is this lib useful on directed graph? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepwalk-c.