Comments (6)
this appeared after I changed a dtype to np.int
. With dtype
back to np.uint16
everything works. I assume that the standard integer
type is not supported on my GPU.
from lattice_lightsheet_deskew_deconv.
I ran into this error again. I can consistenly reproduce it for Stack_7_drp1_dendra2skl_mScarlet_drp1_test_6_fast
. Other datasets process fine, so it may be some corruption of the input data.
/home/vhil0002/Github/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py:207: UserWarning: Fix write_func stuff to include compression and units
warnings.warn("Fix write_func stuff to include compression and units")
/home/vhil0002/Github/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py:262: UserWarning: more than one PSF found. Taking first one
warnings.warn(f"more than one PSF found. Taking first one")
0%| | 0/40 [00:00<?, ?it/s]
Traceback (most recent call last):
File "batch_run.py", line 26, in <module>
ep.process_stack_subfolder(subfolder)
File "/home/vhil0002/Github/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py", line 289, in process_stack_subfolder
self.process_file(pathlib.Path(row.file), deskew_func, rotate_func, deconv_functions[wavelength])
File "/home/vhil0002/Github/Lattice_Lightsheet_Deskew_Deconv/Python/process_llsm_experiment.py", line 198, in process_file
deconv_rotated = rotate_func(deconv_raw)
File "/home/vhil0002/Github/Lattice_Lightsheet_Deskew_Deconv/Python/gputools_wrapper.py", line 76, in affine_transform_gputools
result = gputools.affine(data=input_data, mat=matrix, mode=mode, interpolation=interpolation)
File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/gputools/transforms/transformations.py", line 83, in affine
d_im, res_g.data, mat_inv_g.data)
File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/gputools/core/oclprogram.py", line 46, in run_kernel
self._kernel_dict[name](self._dev.queue,global_size, local_size,*args,**kwargs)
File "/home/vhil0002/anaconda3/envs/newllsm/lib/python3.6/site-packages/pyopencl/__init__.py", line 815, in kernel_call
return self._enqueue(self, queue, global_size, local_size, *args, **kwargs)
File "<generated code>", line 69, in enqueue_knl_affine3
pyopencl._cl.MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE
from lattice_lightsheet_deskew_deconv.
Fairly sure this is happening because tensorflow grabs almost all GPU memory and leaves very little for other processes. Depending on the volume of individual stacks there may be just enough GPU memory left to perform the affine transforms using gputools, but not always. Will have to try limiting how much GPU memory tensorflow can grab:
https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory
using allow_growth might be useful in order not so set a limit automatically https://www.tensorflow.org/guide/using_gpu
This may then also enable several worker processes to use the GPU.
from lattice_lightsheet_deskew_deconv.
fwiw: GPU memory exhaustion errors has a snippet for setting the memory used, and the "allow_growth" option is a good way to avoid tensorflow's very greedy default GPU memory preallocation behavior.
from lattice_lightsheet_deskew_deconv.
Thanks. I had already implemented the allow_growth
fix while on the train home a couple of hours ago. Was just about to test it when I saw your comment. I wasn't aware of the issue you referenced, basically the identical problem ... will put a watch on the flowdec repo.
from lattice_lightsheet_deskew_deconv.
that fixed it, only need to integrate this nicely
from lattice_lightsheet_deskew_deconv.
Related Issues (20)
- Think about deployment for end-users HOT 1
- Real-Time deskew visualization HOT 15
- Add scale to processed tiffs HOT 1
- fixed camera background subtraction - clip at 0, allow negative ? Also include for PSF. HOT 1
- Assertion error in tensorflow
- PSF management - lattice pattern in file name
- include gputools.deconv as deconvolution backend HOT 3
- Performance in batch mode HOT 5
- artefacts introduced by rotate to coverslip HOT 6
- Artefacts on RTX 2080 Ti HOT 1
- Architecture not well-suited for multiprocessing due to pickling limitations HOT 9
- Not enough GPU ram ... make a "slow" option HOT 12
- psf generation - change to two stage affine to avoid artefacts HOT 1
- reuse PSF
- Fail gracefully when out of GPU memory HOT 1
- bug: incompatible shapes HOT 4
- psf z_galvo step extraction fails for 20190802 dataset HOT 1
- CPU version — ModuleNotFoundError: No module named 'numpy.core._multiarray_umath' HOT 6
- Allow deskewing when no PSF is present
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lattice_lightsheet_deskew_deconv.