Comments (7)
Hi modaresimr,
Can you show me the code that produces the memory leak? What OS, Python version, cc3d version are you running?
Thanks for reporting,
Will
from connected-components-3d.
I am using around 1000 time 3d arrays of 512x512x689
int
After 3 use of cc3d, the memory will be increased to 20 GB!!!!!!
It won't release memory!
After a deep debugging by changing that to the code that I mentioned it is fixed.
I will try to take out a part of my code to share soon
from connected-components-3d.
That would be very helpful (along with details such as OS, Python version, and cc3d version). cc3d should be using < 2.2 GB assuming one image in memory.
One thing you can try and see if it helps: cast your array to an unsigned integer type. It shouldn't make a difference, but cc3d was manually tested mainly on unsigned arrays (though integer arrays are included in automated testing).
from connected-components-3d.
Thank you for your support,
I found out that this problem is due to an issue in python memory management when you create a lot of big arrays, and then remove them, the memory will be fragmented, and it can not allocate new arrays.
However, if we create a new process, and run it in a separate process, we will not face any issues.
More info on this issue on: https://stackoverflow.com/a/9617718/834117
from connected-components-3d.
Thanks for looking into this more! If there's a way for you to reuse the arrays (e.g. if they're all the same size) that might help a lot. I'm going to close this issue as it's not directly related to cc3d but is a general python issue. Please reopen if you think you need more help!
from connected-components-3d.
Thanks for looking into this more! If there's a way for you to reuse the arrays (e.g. if they're all the same size) that might help a lot. I'm going to close this issue as it's not directly related to cc3d but is a general python issue. Please reopen if you think you need more help!
Thanks for your suggestion, however, numpy with all the operations create a new array and we can not reuse them :(
from connected-components-3d.
numpy with all the operations create a new array and we can not reuse them :(
It depends on which operations you're doing. For example the +=
operator will avoid creating a copy. You can also check the documentation for several numpy operators which include an out=
parameter. For example, np.multiply(a,b, out=a)
.
from connected-components-3d.
Related Issues (20)
- Applying Dust and largest_k dtype output option HOT 2
- dust sugnature HOT 1
- 1D Array of 4 Elements Incorrect HOT 5
- Cannot find reference 'dust' in 'cc3d.py' HOT 2
- Question on comparing individual lesions between two masks based on the cc3d.statistics output. HOT 1
- Additional metrics support HOT 2
- Does cc3d also work with memmory-mapped numpy arrays and array-like data? HOT 14
- cc3d.statistics["bounding_boxes"] are wrong HOT 1
- largest_k fails for transposed arrays HOT 6
- About the lastest_k function HOT 4
- Statistics output HOT 7
- Question on the output of contacts HOT 9
- Periodic Boundary Conditions HOT 4
- Is the output label of largest_k ordered? HOT 2
- Add a better error for type support. HOT 4
- Any way to make this GPU Compatible? HOT 5
- Applying dust to labels does not do anything HOT 3
- Support for Numpy 2.0 HOT 3
- cc3d.dust fails HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from connected-components-3d.