canonical estimator results in shape error about gluonts

[cyrusmvahid]

I am trying to run the twit example on other estimators. In this case canonical rnn.
Here is the code:

canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
canonical_predictor = canonical_estimator.train(training_data=training_data)

I receive shape error.

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

my context length is 10. It seems as the ndarray has been squeezed somewhere.

Full error:

infer_shape error. Arguments:
  data0: (32, 1)
  data1: (32, 10, 5)
  data2: (32, 10)
---------------------------------------------------------------------------
DeferredInitializationError               Traceback (most recent call last)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args)
    802             cargs = [args[i] if is_arg else i.data()
--> 803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in <listcomp>(.0)
    802             cargs = [args[i] if is_arg else i.data()
--> 803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/parameter.py in data(self, ctx)
    493                                "instead." % (self.name, str(ctx), self._stype))
--> 494         return self._check_and_get(self._data, ctx)
    495 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/parameter.py in _check_and_get(self, arr_list, ctx)
    207                 "You can also avoid deferred initialization by specifying in_units, " \
--> 208                 "num_features, etc., for network layers."%(self.name))
    209         raise RuntimeError(

DeferredInitializationError: Parameter 'rnn8_lstm0_l0_i2h_weight' has not been initialized yet because initialization was deferred. Actual initialization happens during the first forward pass. Please pass one batch of data through the network before accessing Parameters. You can also avoid deferred initialization by specifying in_units, num_features, etc., for network layers.

During handling of the above exception, another exception occurred:

MXNetError                                Traceback (most recent call last)
~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _deferred_infer_shape(self, *args)
    788         try:
--> 789             self.infer_shape(*args)
    790         except Exception as e:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in infer_shape(self, *args)
    861         """Infers shape of Parameters from inputs."""
--> 862         self._infer_attrs('infer_shape', 'shape', *args)
    863 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _infer_attrs(self, infer_fn, attr, *args)
    850             arg_attrs, _, aux_attrs = getattr(out, infer_fn)(
--> 851                 **{i.name: getattr(j, attr) for i, j in zip(inputs, args)})
    852             if arg_attrs is None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py in infer_shape(self, *args, **kwargs)
    995         try:
--> 996             res = self._infer_shape_impl(False, *args, **kwargs)
    997             if res[1] is None:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py in _infer_shape_impl(self, partial, *args, **kwargs)
   1125             ctypes.byref(aux_shape_data),
-> 1126             ctypes.byref(complete)))
   1127         if complete.value != 0:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/base.py in check_call(ret)
    251     if ret != 0:
--> 252         raise MXNetError(py_str(_LIB.MXGetLastError()))
    253 

MXNetError: Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f935a) [0x7f928cf2a35a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f9981) [0x7f928cf2a981]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x73e45d) [0x7f928d26f45d]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x7a25b6) [0x7f928d2d35b6]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x9c33d8) [0x7f928d4f43d8]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e0e10a) [0x7f928f93f10a]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e10a84) [0x7f928f941a84]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x15ba) [0x7f928f8a79aa]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f92fc62dec0]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f92fc62d87d]



During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-143-b6d286edd137> in <module>()
      1 canonical_estimator = CanonicalRNNEstimator(freq="5min", context_length=10, prediction_length=12, trainer=Trainer(epochs=EPOCHS))
----> 2 canonical_predictor = canonical_estimator.train(training_data=training_data)
      3 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluonts/model/estimator.py in train(self, training_data)
    187     def train(self, training_data: Dataset) -> Predictor:
    188 
--> 189         training_transformation, trained_net = self.train_model(training_data)
    190 
    191         # ensure that the prediction network is created within the same MXNet

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluonts/model/estimator.py in train_model(self, training_data)
    180             net=trained_net,
    181             input_names=get_hybrid_forward_input_names(trained_net),
--> 182             train_iter=training_data_loader,
    183         )
    184 

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/gluonts/trainer/_base.py in __call__(self, net, input_names, train_iter)
    256 
    257                             with mx.autograd.record():
--> 258                                 output = net(*inputs)
    259 
    260                                 # network can returns several outputs, the first being always the loss

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in __call__(self, *args)
    538             hook(self, args)
    539 
--> 540         out = self.forward(*args)
    541 
    542         for hook in self._forward_hooks.values():

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in forward(self, x, *args)
    905             with x.context as ctx:
    906                 if self._active:
--> 907                     return self._call_cached_op(x, *args)
    908 
    909                 try:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _call_cached_op(self, *args)
    803                      for is_arg, i in self._cached_op_args]
    804         except DeferredInitializationError:
--> 805             self._deferred_infer_shape(*args)
    806             cargs = []
    807             for is_arg, i in self._cached_op_args:

~/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py in _deferred_infer_shape(self, *args)
    791             error_msg = "Deferred initialization failed because shape"\
    792                         " cannot be inferred. {}".format(e)
--> 793             raise ValueError(error_msg)
    794 
    795     def _call_cached_op(self, *args):

ValueError: Deferred initialization failed because shape cannot be inferred. Error in operator canonicaltrainingnetwork5__minus0: [11:35:55] /work/mxnet/3rdparty/mshadow/../../src/operator/tensor/../elemwise_op_common.h:135: Check failed: assign(&dattr, vec.at(i)) Incompatible attr in node canonicaltrainingnetwork5__minus0 at 1-th input: expected [32,10,1], got [32,10]

Stack trace returned 10 entries:
[bt] (0) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f935a) [0x7f928cf2a35a]
[bt] (1) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x3f9981) [0x7f928cf2a981]
[bt] (2) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x73e45d) [0x7f928d26f45d]
[bt] (3) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x7a25b6) [0x7f928d2d35b6]
[bt] (4) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x9c33d8) [0x7f928d4f43d8]
[bt] (5) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e0e10a) [0x7f928f93f10a]
[bt] (6) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2e10a84) [0x7f928f941a84]
[bt] (7) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x15ba) [0x7f928f8a79aa]
[bt] (8) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f92fc62dec0]
[bt] (9) /home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f92fc62d87d]

[lostella]

@cyrusmvahid thanks for submitting this -- could you enclose snippets and error traces in triple ticks (`) to have it formatted nicely?

like this

[cyrusmvahid]

done
sorry

[lostella]

done
sorry

Thank you, but remember to put triple ticks (```) one line before and one line after snippets and error traces. Sorry for being so pedantic, but nicely formatted issues are more likely to be looked into.

[cyrusmvahid]

sure

…

On Mon, Jul 1, 2019 at 1:47 PM Lorenzo Stella ***@***.***> wrote: done sorry Thank you, but remember to put *triple* ticks (```) one line before and one line after snippets and error traces. Sorry for being so pedantic, but nicely formatted issues are more likely to be looked into. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#161>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABLOVGKEB7UEZ5UDHGYVEXDP5HVFDANCNFSM4H4ROYDA> .

[lostella]

Fixed in #254, closing