TestSPFrequency.testCategory test failure in Vagrant VM about nupic-legacy

[mtbvang]

The entire output of the failure as requested by Matt Taylor:
https://gist.github.com/mtbvang/7458134

[mtbvang]

I ran up my own Lubuntu vagrant box and set it up using the instructions on https://github.com/numenta/nupic/wiki/Install-Nupic-on-ubuntu-13.04 and all the tests passed using code from:

commit e3719ae
Merge: 23a5225 0c36eb9
Author: Matthew Taylor [email protected]
Date: Tue Nov 12 11:00:00 2013 -0800

The code that failed with the Numenta VM was from:

commit 23a5225
Merge: 5fe9e3e 4abef3c
Author: Matthew Taylor [email protected]
Date: Fri Nov 8 11:39:45 2013 -0800

I hope this isn't a red herring caused by it being my first time environment setup.

My system is:

Lubuntu 13.10 64bit
Virtualbox 4.3.2
Vagrant 1.3.5

[breznak]

Hi,

I don't understand what you''re observing? You say all tests pass oK with e3719ae from Nov 12 (= master/HEAD), but then what does the 23a5225 from Nov 8 mean?

Or did you figure 23a5225 is the offending commit? (shouldn't be) - try git revert 23a5225

[mtbvang]

Breznak, on the 8 Nov I tried runniing up the Numenta VM and got the test failures. Since then I have also run up my own installation on Lubuntu using the wiki/Install-Nupic-on-ubuntu-13.04 intructions and all the test ran fine on that. I haven't gone back to try and get the Numenta VM since the test error.

[breznak]

Ah, I see. So changing the title. Glad it runs for you somehow, so it's not a hot issue. Feel free to comment on the bug if you do retest that, or just close if you dont bother anymore.

[mrcslws]

I'm hitting this same error. I hit it with the unit tests
$NUPIC/run_tests.sh
and with the example
python $NUPIC/examples/opf/bin/OpfRunExperiment.py $NUPIC/examples/opf/experiments/multistep/hotgym/

The common error is:
TypeError: Cannot cast array data from dtype('float32') to dtype('<m8') according to the rule 'safe'

More fully:
[vagrant@nupic-vagrant:/vagrant]$ python $NUPIC/examples/opf/bin/OpfRunExperiment.py $NUPIC/examples/opf/experiments/multistep/hotgym/ Using logging configuration file: /home/vagrant/nta/eng/conf/default/nupic-logging.conf Traceback (most recent call last): File "/vagrant/examples/opf/bin/OpfRunExperiment.py", line 51, in <module> main() File "/vagrant/examples/opf/bin/OpfRunExperiment.py", line 46, in main runExperiment(sys.argv[1:]) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/experiment_runner.py", line 147, in runExperiment model = _runExperimentImpl(opt, model) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/experiment_runner.py", line 409, in _runExperimentImpl model = ModelFactory.create(modelDescription) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/modelfactory.py", line 76, in create return modelClass(**modelConfig['modelParams']) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/clamodel.py", line 203, in __init__ anomalyParams) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/frameworks/opf/clamodel.py", line 1200, in __createCLANetwork n.initialize() File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/engine/__init__.py", line 694, in initialize engine.Network.initialize(self, *args, **kwargs) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/bindings/engine_internal.py", line 1034, in initialize return _engine_internal.Network_initialize(*args, **kwargs) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/regions/SPRegion.py", line 459, in initialize self._allocateSpatialFDR(None) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/regions/SPRegion.py", line 516, in _allocateSpatialFDR **autoArgs) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/research/FDRCSpatial2.py", line 449, in __init__ self.random)) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/research/FDRCSpatial2.py", line 1342, in _makeMasterCoincidences masterPotentialM.append(SM_01_32_32(denseM.reshape(coincRFShape))) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/bindings/math.py", line 4572, in __init__ self.fromDense(numpy.asarray(args[0])) File "/home/vagrant/nta/eng/lib/python2.6/site-packages/nupic/bindings/math.py", line 4695, in fromDense return _math._SM_01_32_32_fromDense(*args, **kwargs) TypeError: Cannot cast array data from dtype('float32') to dtype('<m8') according to the rule 'safe'

This is with the Numenta VM from https://github.com/numenta/nupic/wiki/Running-Nupic-in-a-Virtual-Machine running on VirtualBox running on Windows.

I haven't successfully worked around the issue.

[mrcslws]

I did some debugging. I plan on continuing, but here's what I learned.

Here's a gdb snippet that shows where this is going very badly:

3877        if (type < NPY_NTYPES) {
(gdb) n
3878            ret = _builtin_descrs[type];
(gdb) p type
$31 = 22
(gdb) backtrace
#0  PyArray_DescrFromType (type=22)
    at numpy/core/src/multiarray/arraytypes.c.src:3878
#1  0x00007fa3d3da8abb in nta::NumpyArray::NumpyArray (this=0x7ffff08b2860,
    p=0x2e89350, dtype=6, requiredDimension=2)
    at /vagrant/lang/py/support/NumpyVector.cpp:147
#2  0x00007fa3d3c50e24 in NumpyMatrixT (args=<value optimized out>,
    kwargs=<value optimized out>) at /vagrant/lang/py/support/NumpyVector.hpp:236
#3  nta_SparseBinaryMatrix_Sl_nta_UInt32_Sc_nta_UInt32_Sg__fromDense (
    args=<value optimized out>, kwargs=<value optimized out>)
    at /tmp/ntabuild/py/bindings/math/math_pyPYTHON_wrap.cxx:6665
#4  _wrap__SM_01_32_32_fromDense (args=<value optimized out>,
    kwargs=<value optimized out>)
    at /tmp/ntabuild/py/bindings/math/math_pyPYTHON_wrap.cxx:37721

If you look around, this should never happen. This function is called with type == PyArray_NOTYPE, i.e. NPY_NOTYPE, i.e. the value in the enum that comes after NPY_NTYPES. So it should not have entered into this if (type < NPY_NTYPES) block.

Here's the enum in question.

enum NPY_TYPES {    NPY_BOOL=0,
                    NPY_BYTE, NPY_UBYTE,
                    NPY_SHORT, NPY_USHORT,
                    NPY_INT, NPY_UINT,
                    NPY_LONG, NPY_ULONG,
                    NPY_LONGLONG, NPY_ULONGLONG,
                    NPY_FLOAT, NPY_DOUBLE, NPY_LONGDOUBLE,
                    NPY_CFLOAT, NPY_CDOUBLE, NPY_CLONGDOUBLE,
                    NPY_OBJECT=17,
                    NPY_STRING, NPY_UNICODE,
                    NPY_VOID,
                    /*
                     * New 1.6 types appended, may be integrated
                     * into the above in 2.0.
                     */
                    NPY_DATETIME, NPY_TIMEDELTA, NPY_HALF,

                    NPY_NTYPES,
                    NPY_NOTYPE,
                    NPY_CHAR,      /* special flag */
                    NPY_USERDEF=256,  /* leave room for characters */

                    /* The number of types not including the new 1.6 types */
                    NPY_NTYPES_ABI_COMPATIBLE=21
};

So /vagrant/lang/py/support/NumpyVector.cpp thinks that NPY_NOTYPE is 22, eh? Well, multiarray.so thinks that 22 corresponds to NPY_TIMEDELTA. You can see how this would lead to this error message. dtype('<m8') is a DateTime datatype.

So the binaries are using different versions of an enum. I'll try to find where it's pulling the stale enum value...

[mrcslws]

Interesting, the issue stopped reproing after I specifically ran:
make lang/py/support/NumpyVector.o
make support
make _math/fast
from $NUPIC/build_system (after editing $NUPIC/lang/py/support/NumpyVector.cpp to mark the file as dirty). Now when I step through, it uses enum value 25 instead of 22.

I might be pinpointing this incorrectly -- I ran a few other build commands today not mentioned above. But my point is that this seems to be a subtle build-ordering issue.

[mrcslws]

...and now I synced to latest, and it's not reproing. Not even on a fresh VM. My previous comment is still correct -- I went from repro -> no repro without changing/syncing any code. Anyway, maybe something from the past week fixed the issue.

[rhyolight]

@mtbvang Can you pull the latest, rebuild and try running tests again? I'd like to see if the problem is still occurring in your environment.