"nosetests -x -s -v tests" returns cufftInvalidValue (error in test_fft_float64_to_complex128) about scikit-cuda

[lebedov]

I can't reproduce this consistently with the latest revision of scikits.cuda (as of the time of this reply) on a 64-bit Linux system running CUDA 3.2 and PyCUDA 0.94.2; I noticed only one occurrence of the error when I ran the test multiple times on different machines running the same software.

[npinto]

There might be some race condition happening... Or problems with memory, have you run it through cuda-memcheck ?

[lebedov]

Hmm..Running the following code through cuda-gdb on a system containing a compute capability 2.1 device 0 and a compute capability 1.1 device 1 seems to cause the cufftInvalidValue exception to occur consistently in the cufftPlanMany function (regardless of whether memcheck is activated):

import pycuda.autoinit
import pycuda.gpuarray as gpuarray
import numpy as np
import scikits.cuda.cufft as cufft

N = 128
x = np.asarray(np.random.rand(N), np.float64)
xf = np.fft.fft(x)
x_gpu = gpuarray.to_gpu(x)
xf_gpu = gpuarray.empty(N/2+1, np.complex128)
n = np.asarray(x_gpu.shape, np.int32)
handle = cufft.cufftPlanMany(1, n.ctypes.data, cufft.CUFFT_D2Z, 1)
cufft.cufftExecD2Z(handle, int(x_gpu.gpudata), int(xf_gpu.gpudata))

Does your system contain any GPUs that don't support double precision?
(By the way, I'm assuming that the only way to run cuda-memcheck on a PyCUDA program is through cuda-gdb; feel free to correct me if I am wrong.)

[larsoner]

Hey @lebedov, just wanted to let you know that I hit this error on my laptop, I think you are right it's due to not supporting 64-bit float (double) operations, at least for FFTs. Let me know if you develop some code that should check for the capability and possibly throw a more informative error, and I can try it out.

[lebedov]

I just added a check to the Plan class constructor to check for the appropriate compute capability before attempting to process double precision arguments.

[larsoner]

I updated my master to the latest version, but unfortunately my system doesn't have CULA installed, and now I get an error when trying to import scikits.cuda.fft:

>>> from scikits.cuda import fft as cudafft
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/fft.py", line 19, in <module>
    import misc
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/misc.py", line 20, in <module>
    import cula
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/cula.py", line 32, in <module>
    raise OSError('%s not found' % _load_err)
OSError: libcula_lapack.so, libcula.dylib not found

Is there any way to remove this new requirement that libcula be installed? I don't have it installed on either my OSX install (this machine that lacks 64-bit computation support) or on my Ubuntu 64-bit workstation (with 64-bit support). I had been using a version of scikits.cuda from only a couple months ago (December, maybe?) that didn't throw this error / have this requirement, so it must have been some relatively recent change that added this requirement when importing scikits.cuda.fft.

I tried installing CULA on each of my machines at one point and found it to be more hassle than it was worth, given that I could use the fft module without it. But if I have to do it in order to get CUDA working again with the latest version, I'll figure it out...

[larsoner]

In case you were wondering, this is how the old version would break -- note that the import worked, but I got the cufftInvalidValue error that was mentioned before only when doing float64 FFTs:

>>> import pycuda.autoinit
>>> from pycuda import gpuarray
>>> from scikits.cuda import fft as cudafft
>>> float32_plan = cudafft.Plan(16, np.float32, np.complex64)
>>> float64_plan = cudafft.Plan(16, np.float64, np.complex128)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/fft.py", line 85, in __init__
    self.fft_type, self.batch)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/cufft.py", line 179, in cufftPlanMany
    cufftCheckStatus(status)
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/cufft.py", line 91, in cufftCheckStatus
    raise cufftExceptions[status]
scikits.cuda.cufft.cufftInvalidValue

I'll update with what happens with the newer version once I get CULA installed (which might be a nightmare since I'm running 32-bit python on 64-bit OSX) or once the CULA requirement is removed from scikits.cuda.fft.

[larsoner]

Oh -- looks like all you might need to do is instead of except ImportError: just do the more general except (ImportError, OSError): (or be sloppy and do except Exception: on line 22 of misc.py, since my system is throwing an OSError instead of ImportError when it tries to load CULA.

[larsoner]

With the fix from my PR, I now get this:

>>> import pycuda.autoinit
>>> from scikits.cuda import fft as cudafft
>>> float32_plan = cudafft.Plan(16, np.float32, np.complex64)
>>> float64_plan = cudafft.Plan(16, np.float64, np.complex128)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site-packages/scikits.cuda-0.042-py2.7.egg/scikits/cuda/fft.py", line 85, in __init__
    raise RuntimeError('double precision requires compute capability >= 1.3')
RuntimeError: double precision requires compute capability >= 1.3

Hooray!

[larsoner]

In my PR I also increased the informativity of the error to:

RuntimeError: double precision requires compute capability >= 1.3 (you have 1.2)

I hope that's alright -- otherwise it's easy enough to close my PR and just change the one import-breaking line :)

[lebedov]

Thanks! Merged.

[larsoner]

No problem. Thanks for a great project -- using CUDA is a few times faster than filtering / resampling using the CPU, and a project I'm working on does a lot of filtering.

[lebedov]

Closing old issue.