dmlc / mxnet-notebooks

Notebooks for MXNet

Distributed Machine Learning Common Codebase

DMLC-Core is the backbone library to support all DMLC projects, offers the bricks to build efficient and scalable distributed machine learning libraries.

Developer Channel

What's New

• Note on Parameter Module for Machine Learning

Contents

• Documentation and Tutorials
• Contributing

Known Issues

• RecordIO format is not portable across different processor endians. So it is not possible to save RecordIO file on a x86 machine and then load it on a SPARC machine, because x86 is little endian while SPARC is big endian.

Contributing

Contributing to dmlc-core is welcomed! dmlc-core follows google's C style guide. If you are interested in contributing, take a look at feature wishlist and open a new issue if you like to add something.

• DMLC-Core uses C++11 standard. Ensure that your C++ compiler supports C++11.
• Try to introduce minimum dependency when possible

CheckList before submit code

• Type make lint and fix all the style problems.
• Type make doc and fix all the warnings.

NOTE

deps:

libcurl4-openssl-dev

mxnet-notebooks's Issues

matrix factorization notebook, no activations?

What is the benefit of adding in another layer if you're using a linear activation? My understanding was that multiple linear layers can be expressed as a single linear layer.

Thanks!

Error when change num_hidden = 256

In using char_lstm.ipyn I changed num_hidden from 512 to 256, and the model training finished without errors. But I got the following error message when I tried to make the inference model using the following command.

model_inf = rnn_model.LSTMInferenceModel(
num_lstm_layer,
len(vocab) + 1,
num_hidden=num_hidden,
num_embed=num_embed,
num_label=len(vocab) + 1,
arg_params=arg_params,
ctx=mx.gpu(),
dropout=0.2)

[22:45:14] /home/ec2-user/src/mxnet/dmlc-core/include/dmlc/logging.h:235: [22:45:14] src/ndarray/ndarray.cc:231: Check failed: from.shape() == to->shape() operands shape mismatch
Traceback (most recent call last):
File "", line 10, in
File "/home/ec2-user/mxnet-notebooks/python/tutorials1/rnn_model.py", line 34, in init
arg_params[key].copyto(self.executor.arg_dict[key])
File "/usr/local/lib/python2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/ndarray.py", line 533, in copyto
return _internal._copyto(self, out=other)
File "/usr/local/lib/python2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/ndarray.py", line 1225, in unary_ndarray_function
c_array(ctypes.c_char_p, [c_str(str(i)) for i in kwargs.values()])))
File "/usr/local/lib/python2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/base.py", line 77, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [22:45:14] src/ndarray/ndarray.cc:231: Check failed: from.shape() == to->shape() operands shape mismatch

Can't test the online classify of LeNet in python/tutorials/mnist.ipynb

Found that the sample python/tutorials/mnist.ipynb include codes of MLP and LeNet, while the model of LeNet can't be used in the HTML online classify cell.

I would like to suggest that we can create two mnist demo, one is basic and the other one is advance.

In the basic one , say mnist_basic.ipynb, we can demo the MLP network with CPU, in the advice one , say mnist_advance.ipynb, we can demo the LeNet network with GPU.

If it is OK, I will create a pull request for that . Looking forward to your response.

Is This Project Still Being Maintained?

Is this project still being maintained? The latest update seems to be about two years ago.

ImportError: No module named get_data in cifar10

import sys
sys.path.append("../../tests/python/common")
import get_data

ImportError Traceback (most recent call last)
in ()
1 import sys
2 sys.path.append("../../tests/python/common")
----> 3 import get_data

ImportError: No module named get_data

dimension mismatch in LSTM tutorial when num_embed != num_hidden

LSTM tutorial (https://github.com/dmlc/mxnet-notebooks/blob/master/python/rnn/lstm.ipynb) seems to have a bug which does not support num_embed != num_hidden. next_state in lstm() should be defined like below.

            next_state = lstm(num_embed if i == 0 else num_hidden, indata=hidden,
                              mask=maskvec[seqidx],
                              prev_state=last_states[i],
                              param=param_cells[i],
                              seqidx=seqidx, layeridx=i, dropout=dropout)

Below is an expanded context:

embed = mx.sym.Embedding(data=data, input_dim=input_size,
                             weight=embed_weight, output_dim=num_embed, name='embed')
    wordvec = mx.sym.SliceChannel(data=embed, num_outputs=seq_len, squeeze_axis=1)
    maskvec = mx.sym.SliceChannel(data=mask, num_outputs=seq_len, squeeze_axis=1)

    # Now we can unroll the network
    hidden_all = []
    for seqidx in range(seq_len):
        hidden = wordvec[seqidx] # input to LSTM cell, comes from embedding

        # stack LSTM
        for i in range(num_lstm_layer):
            next_state = lstm(num_hidden, indata=hidden,
                              mask=maskvec[seqidx],
                              prev_state=last_states[i],
                              param=param_cells[i],
                              seqidx=seqidx, layeridx=i, dropout=dropout)
            hidden = next_state.h
            last_states[i] = next_state
        # decoder
        hidden_all.append(hidden) # last output of stack LSTM units

Images do not render in LSTM tutorial

In https://github.com/dmlc/mxnet-notebooks/blob/master/python/rnn/lstm.ipynb, some of the images that at in the source code and are referenced in the text do not render. Specifically, https://raw.githubusercontent.com/antinucleon/web-data/master/mxnet/notebook/recurrent.jpg and
https://raw.githubusercontent.com/antinucleon/web-data/master/mxnet/notebook/unroll.jpg do not render. Both of these links give 404s.

Do we need to subtract mean pixel values when extracting cnn feautre

Hi! I followed this notebook to extract image feature using pre-trained CNN model. I wonder do we need to subtract mean pixel values of imagenet before feature extraction. Other deep learning frameworks usually have this process, but I didn't find it in this notebook.

AttributeError: 'module' object has no attribute 'sum_axis'

Notebooks data.ipynb and matrix_factorization.ipynb are populated with

mx.sym.sum_axis()

which raises the attribute error.

Guess should be

mx.sym.sum()

Update jupyter notebook tutorials to use Module interface?

When I run the the jupyter notebook tutorials, it said that:
DeprecationWarning: mxnet.model.FeedForward has been deprecated. Please use mxnet.mod.Module instead.
So I hope these jupyter notebook tutorials will be updated to use the Module interface.

Issue in rnn/lstm.ipynb

In cell No.7

sym = lstm_unroll(num_lstm_layer=num_lstm_layer,
                  seq_len=seq_len,
                  input_size=vocab_size,
                  num_hidden=num_hidden,
                  num_embed=num_embed,
                  num_label=vocab_size,
                  ignore_label=0)

AttributeError: 'module' object has no attribute 'element_mask'

@aileli

ImportError: No module named data_iter in module.ipynb

from data_iter import SyntheticData

However, I got "ImportError: No module named data_iter". Sorry that I got little knowledge on " data_iter" and "SyntheticData".

Should I import some dependancies/libs first?

Can't import tensorboard summary and FileWriter

In https://github.com/dmlc/mxnet-notebooks/blob/master/python/tutorials/understanding_vanish_gradient.ipynb:

from tensorboard import summary
from tensorboard import FileWriter

Fails

I tested it on Amazon Ubuntu.
@zihaolucky

Error when running collaborative-dl.ipynb

Traceback:

AttributeError Traceback (most recent call last)
in ()
24 fp.close()
25
---> 26 cal_rec(1,8)

in cal_rec(p, cut)
17 l_rec = list(zip(*pl)[0])[:cut]
18 s_rec = set(l_rec)
---> 19 s_true = set(np.where(R_true[i,:]>0)[1].A1)
20 cnt_hit = len(s_rec.intersection(s_true))
21 fp.write('%d:' % cnt_hit)

AttributeError: 'numpy.ndarray' object has no attribute 'A1'

@aileli

Resnet18 is predicting incorrect label

I wrote a following script to predict image label for resent 18.

import mxnet as mx
model_name = 'resnet-18'
path='http://data.mxnet.io/models/imagenet/resnet/'
    [mx.test_utils.download(path+'18-layers/resnet-18-symbol.json'),
     mx.test_utils.download(path+'18-layers/resnet-18-0000.params'),
     mx.test_utils.download(path+'synset.txt')]
sym, arg_params, aux_params = mx.model.load_checkpoint(model_name, 0)
mod = mx.mod.Module(symbol=sym, context=mx.cpu(), label_names=None)
mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))], 
         label_shapes=mod._label_shapes)
mod.set_params(arg_params, aux_params, allow_missing=True)
with open('synset.txt', 'r') as f:
    labels = [l.rstrip() for l in f]

%matplotlib inline
import matplotlib.pyplot as plt
import cv2
import numpy as np
# define a simple data batch
from collections import namedtuple
Batch = namedtuple('Batch', ['data'])

def get_image(url, show=False):
    # download and show the image
    fname = mx.test_utils.download(url)
    img = cv2.cvtColor(cv2.imread(fname), cv2.COLOR_BGR2RGB)
    if img is None:
         return None
    if show:
         plt.imshow(img)
         plt.axis('off')
    # convert into format (batch, RGB, width, height)
    img = cv2.resize(img, (224, 224))
    img = np.swapaxes(img, 0, 2)
    img = np.swapaxes(img, 1, 2)
    img = img[np.newaxis, :]
    return img

def predict(url):
    img = get_image(url, show=True)
    # compute the predict probabilities
    mod.forward(Batch([mx.nd.array(img)]))
    prob = mod.get_outputs()[0].asnumpy()
    # print the top-5
    prob = np.squeeze(prob)
    a = np.argsort(prob)[::-1]
    for i in a[0:5]:
        print('probability=%f, class=%s' %(prob[i], labels[i]))

predict('http://writm.com/wp-content/uploads/2016/08/Cat-hd-wallpapers.jpg')

Output:

probability=0.244390, class=n01514668 cock
probability=0.170342, class=n01514752 gamecock, fighting cock
probability=0.145019, class=n01495493 angel shark, angelfish, Squatina squatina, monkfish
probability=0.059832, class=n01540233 grosbeak, grossbeak
probability=0.051555, class=n01517966 carinate, carinate bird, flying bird

I am getting completly wrong output.

Same code I tried with Resent 152., Output I got is correct

probability=0.692327, class=n02122948 kitten, kitty
probability=0.043847, class=n01323155 kit
probability=0.030002, class=n01318894 pet
probability=0.029693, class=n02122878 tabby, queen
probability=0.026972, class=n01322221 baby

Issues in IO Image Notebooks

Some issues in io image notebooks:

1. In record_io.ipynb cell No.9:

s = mx.recordio.pack_img(header, data, quality=100, img_fmt='.jpg')

results in segmentation fault.

2. In advanced_img_io.ipynb cell No.9:

img = cv2.imread('test_images/ILSVRC2012_val_00000001.JPEG', flags=1)

results in segmentation fault. I searched this online and found it might be an opencv bug.

3. In image_io.ipynb cell No.5:
   MXNetError: [07:28:27] src/io/input_split_base.cc:163: Check failed: files_.size() != 0U (0 vs. 0) Cannot find any files that matches the URI patternz ./data/caltech_train.rec

All these tests were run on AMI Ubuntu. I also rebuilt mxnet using updated github version.
@piiswrong Any idea?

Some question about finetuning

1. During fine-tuning, is it possible to set some layers' learning rate as 0, if possible, how to do?
2. In multi-task fine-tuning, how to alter multi-losses with the new one?
3. If I want to remove the last several layers, and replace them with new layers, how to implement?
   Thanks a lot!

How to run these ipynb example?

There is no document how to run these examples?

deep matrix factorization question

The code currently looks like this:

def get_one_layer_mlp(hidden, k):
    # input
    user = mx.symbol.Variable('user')
    item = mx.symbol.Variable('item')
    score = mx.symbol.Variable('score')
    # user latent features
    user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)
    user = mx.symbol.Activation(data = user, act_type="relu")
    user = mx.symbol.FullyConnected(data = user, num_hidden = hidden)
    # item latent features
    item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)
    item = mx.symbol.Activation(data = item, act_type="relu")
    item = mx.symbol.FullyConnected(data = item, num_hidden = hidden)
    # predict by the inner product
    pred = user * item
    pred = mx.symbol.sum_axis(data = pred, axis = 1)
    pred = mx.symbol.Flatten(data = pred)
    # loss layer
    pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)
    return pred

My understanding is that the embedding layer should be able to learn anything that having a single dense layer on top of it could learn, since it can be basically anything. I had thought a deep matrix factorization would look something more like this:

def get_one_layer_mlp(hidden, k):
    # input
    user = mx.symbol.Variable('user')
    item = mx.symbol.Variable('item')
    score = mx.symbol.Variable('score')
    # user latent features
    user = mx.symbol.Embedding(data = user, input_dim = max_user, output_dim = k)

    # item latent features
    item = mx.symbol.Embedding(data = item, input_dim = max_item, output_dim = k)

    # predict by the inner product
    pred = mx.symbol.Concat([user, item])
    pred = mx.symbol.FullyConnected(data = pred, num_hidden = hidden)
    pred = mx.symbol.Activation(data = pred, act_type="relu")
    pred = mx.symbol.FullyConnected(data = pred, num_hidden = 1)

    # loss layer
    pred = mx.symbol.LinearRegressionOutput(data = pred, label = score)
    return pred

Basically, the layers should take a concatenation of the latent variables and have layers on top of that, instead of having layers on top of the embedding layers.

lstm_bucketing tutorial can't run

Missing ptb.train.txt file.
I found the get_ptb_data.sh in mxnet/example/rnn directory so it should be fine.

But the lstm_bucketing.ipynb missed several png files?

update2: the lstm_bucketing.py in mxnet/example/rnn could run. But the notebooks version can't run.

the module don't predict the single sample python/basic/module.ipynb???

I want to predict the single sample or some samples by using the module.
the module provides the predict function, But it had to carry a data iterators as arguments.

LinearRegression Example

Bug Report

ZeroDivisionError Traceback (most recent call last)
in ()
38 optimizer_params={'learning_rate':0.01, 'momentum': 0.9},
39 num_epoch=1000,
---> 40 batch_end_callback = mx.callback.Speedometer(batch_size, 1))
41
42

/auto/homes/yn272/.virtualenvs/HDR/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/base_module.pyc in fit(self, train_data, eval_data, eval_metric, epoch_end_callback, batch_end_callback, kvstore, optimizer, optimizer_params, eval_end_callback, eval_batch_end_callback, initializer, arg_params, aux_params, allow_missing, force_rebind, force_init, begin_epoch, num_epoch, validation_metric, monitor)
420 locals=locals())
421 for callback in _as_list(batch_end_callback):
--> 422 callback(batch_end_params)
423
424 # one epoch of training is finished

/auto/homes/yn272/.virtualenvs/HDR/local/lib/python2.7/site-packages/mxnet-0.9.4-py2.7.egg/mxnet/callback.pyc in call(self, param)
114 if self.init:
115 if count % self.frequent == 0:
--> 116 speed = self.frequent * self.batch_size / (time.time() - self.tic)
117 if param.eval_metric is not None:
118 name_value = param.eval_metric.get_name_value()

ZeroDivisionError: float division by zero

linear-regression.ipynb returns 'nan' score

In linear-regression.ipynb cell No.9 and No.10, "model.score(eval_iter, metric)" occasionally return 'nan' as mean square error.

Tested on docker ubuntu with mxnet installed with 'pip install mxnet-cu75'.

Can you reproduce this issue?
@karishmamalkan

single place for downloading required data

can all required data be downloaded to a single place for all jupyter notebooks?
it is not fast to download data using the python script in the notebook.
if I want to use GPU on my own computer, I have to wait it downloads data each time...

How is the obama-0075.params file generated

I've had a look at the char_lstm tutorial and notice a file named obama-0075.params that is included in the zip file download.

Could you explain how this file is generated or point me in the right direction?

Sample codes to work on ConvLSTM and ConvGRU?

Iam learning to work on ConvLSTM and ConvGRU. Kindly suggest some examples to work on these modules

notebooks should be more beginner friendly

the notebooks should be grouped into three levels

• for beginners: basic usages for how to define (symbol) and train (module) a neural network.
• for power users: more mechanisms, such as lazy evaluation, auto parallelization, advanced usages
• for hackers: how to implement customized operator/optimizer...

Kernel issue for new notebook

In basic folder, three new notebooks are added: advanced_img_io.ipynb, image_io.ipynb, record_io.ipynb. But the kernel of these three notebooks is set as 'Python [Root]', which can't be recognized by Jupyter notebook.

@piiswrong

ndarray.ipynb doesn't load

Getting error "Sorry, something went wrong. Reload?" with no improvement after reloading

Error when run char_lstm example

I run the char_lstm example on my mbp. As the computer has no NVIDIA card，so I change the notebook to use cpu(0), the change is at below:

model = mx.model.FeedForward(
ctx=mx.cpu(0),//chang ctx from mx.gpu(0)to mx.cpu(0)
symbol=symbol,
num_epoch=num_epoch,
learning_rate=learning_rate,
momentum=0,
wd=0.0001,
initializer=mx.init.Xavier(factor_type="in", magnitude=2.34))

and when I run the cell, the compiler report errors:

---

TypeError Traceback (most recent call last)
in ()
38 eval_metric=mx.metric.np(Perplexity),
39 batch_end_callback=mx.callback.Speedometer(batch_size, 20),
---> 40 epoch_end_callback=mx.callback.do_checkpoint("obama"))

/Library/Python/2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/model.pyc in fit(self, X, y, eval_data, eval_metric, epoch_end_callback, batch_end_callback, kvstore, logger, work_load_list, monitor, eval_batch_end_callback)
786 logger=logger, work_load_list=work_load_list, monitor=monitor,
787 eval_batch_end_callback=eval_batch_end_callback,
--> 788 sym_gen=self.sym_gen)
789
790

/Library/Python/2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/model.pyc in _train_multi_device(symbol, ctx, arg_names, param_names, aux_names, arg_params, aux_params, begin_epoch, end_epoch, epoch_size, optimizer, kvstore, update_on_kvstore, train_data, eval_data, eval_metric, epoch_end_callback, batch_end_callback, logger, work_load_list, monitor, eval_batch_end_callback, sym_gen)
243
244 # evaluate at end, so we can lazy copy
--> 245 executor_manager.update_metric(eval_metric, data_batch.label)
246
247 nbatch += 1

/Library/Python/2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/executor_manager.pyc in update_metric(self, metric, labels)
404 def update_metric(self, metric, labels):
405 """update metric with the current executor"""
--> 406 self.curr_execgrp.update_metric(metric, labels)

/Library/Python/2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/executor_manager.pyc in update_metric(self, metric, labels)
260 for texec, islice in zip(self.train_execs, self.slices):
261 labels_slice = [label[islice] for label in labels]
--> 262 metric.update(labels_slice, texec.outputs)
263
264 class DataParallelExecutorManager(object):

/Library/Python/2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/metric.pyc in update(self, labels, preds)
342 pred = pred[:, 1]
343
--> 344 reval = self._feval(label, pred)
345 if isinstance(reval, tuple):
346 (sum_metric, num_inst) = reval

/Library/Python/2.7/site-packages/mxnet-0.7.0-py2.7.egg/mxnet/metric.pyc in feval(label, pred)
368 def feval(label, pred):
369 """Internal eval function."""
--> 370 return numpy_feval(label, pred)
371 feval.name = numpy_feval.name
372 return CustomMetric(feval, name, allow_extra_outputs)

in Perplexity(label, pred)
23
24
---> 25 loss += -np.log(max(1e-10, pred[i][int(label[i])]))
26 return np.exp(loss / label.size)
27

TypeError: only length-1 arrays can be converted to Python scalars

the error arise from the code ：
loss += -np.log(max(1e-10, pred[i][int(label[i])]))

so is it caused by the ctx change? please help

Install package for notebook

In some notebooks, there are some importing modules need to be installed before user can run the notebook. We can:
(1) add all those modules as mxnet installation dependencies. This may require system maintainer to frequently update mxnet dependencies when we have new notebooks.
(2) Use pip inside notebook to install dependencies while running notebook, This may require notebook writer to identify the modules need to be installed.

@mli @piiswrong

simple_test for new inception-bn pretrained

import mxnet as mx
ctx = mx.gpu(1)

val = mx.io.ImageRecordIter(
    path_imgrec = 'val_256_q90.rec',
    mean_r = 123.68,
    mean_g = 116.779,
    mean_b = 103.939,
    rand_crop = False,
    rand_mirror = False,
    data_shape = (3, 224, 224),
    batch_size = 32,
    num_parts = 1,
    part_index = 0)
sym, args, auxs = mx.model.load_checkpoint('Inception-BN', 126)
model = mx.model.FeedForward(sym, ctx=ctx, arg_params=args, aux_params=auxs)
score = model.score(val)
print score

[13:50:36] src/io/iter_image_recordio.cc:209: ImageRecordIOParser: val_256_q90.rec, use 4 threads for decoding..
0.724528150992

Install on Mac OS X with the script for Ubuntu?

How to install mxnet-notebooks on Mac OS X, can I just us the example script proved for Ubuntu to do the installation?

Thanks

Here is an example script to install all these packages on Ubuntu