Premature stopping of the inference of the example nntool/mnist. about gap_sdk

[Pafrak]

Hello,
no we never noticed it, can you provide the configuration for cases in which it stops prematurely?

[Olivier-Dopeli]

I've modified the train.py code below. I just commented one layer.

#!/usr/bin/python3.6
# PYTHON_ARGCOMPLETE_OK

# Copyright (C) 2019 GreenWaves Technologies
# All rights reserved.

# This software may be modified and distributed under the terms
# of the BSD license.  See the LICENSE file for details.

'''Trains a simple convnet on the MNIST dataset.
Gets to 99.25% test accuracy after 12 epochs
(there is still a lot of margin for parameter tuning).
16 seconds per epoch on a GRID K520 GPU.
'''

from __future__ import print_function

import argparse

import argcomplete
import keras
from keras import backend as K
from keras.datasets import mnist
from keras.layers import Activation, Conv2D, Dense, Flatten, MaxPooling2D, BatchNormalization
from keras.models import Sequential
import time

def create_parser():
    # create the top-level parser
    parser = argparse.ArgumentParser(prog='train')

    parser.add_argument('h5_file',
                        default="output.h5",
                        nargs=argparse.OPTIONAL,
                        help='Output - Trained model in h5 format')
    parser.add_argument('-b', '--batch_size',
                        type=int,
                        default=128,
                        help='training batch size')
    parser.add_argument('-e', '--epochs',
                        type=int,
                        default=5,
                        help='training epochs')
    parser.add_argument('-c', '--clip',
                        action='store_true',
                        help='clip input to 7 bits')
    parser.add_argument('-B', '--batch_norm',
                        action='store_true',
                        help='carry out batch normalization')
    return parser

def train(args):
    batch_size = args.batch_size
    num_classes = 10
    epochs = args.epochs

    # input image dimensions
    img_rows, img_cols = 28, 28

    # the data, split between train and test sets
    (x_train, y_train), (x_test, y_test) = mnist.load_data()

    if K.image_data_format() == 'channels_first':
        x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
        x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
        input_shape = (1, img_rows, img_cols)
    else:
        x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
        x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
        input_shape = (img_rows, img_cols, 1)

    if args.clip:
        x_train >>= 1
        x_test >>= 1

    x_train = x_train.astype('float32')
    x_test = x_test.astype('float32')

    if args.clip:
        x_train /= 127
        x_test /= 127
    else:
        x_train /= 255
        x_test /= 255
    print('x_train shape:', x_train.shape)
    print(x_train.shape[0], 'train samples')
    print(x_test.shape[0], 'test samples')

    # convert class vectors to binary class matrices
    y_train = keras.utils.to_categorical(y_train, num_classes)
    y_test = keras.utils.to_categorical(y_test, num_classes)

    model = Sequential()

    model.add(Conv2D(32, kernel_size=(5, 5), input_shape=input_shape))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
     
    #model.add(Conv2D(128, (5, 5)))
    #if args.batch_norm:
    #    model.add(BatchNormalization())
    #model.add(Activation('relu'))

    model.add(Flatten())
    #model.add(Flatten(data_format='channels_first'))
    model.add(Dense(num_classes))
    if args.batch_norm:
        model.add(BatchNormalization())
    model.add(Activation('softmax'))

    model.compile(loss=keras.losses.categorical_crossentropy,
                  optimizer=keras.optimizers.Adadelta(),
                  metrics=['accuracy'])

    model.fit(x_train, y_train,
              batch_size=batch_size,
              epochs=epochs,
              verbose=1,
              validation_data=(x_test, y_test))
    score = model.evaluate(x_test, y_test, verbose=0)
    print('Test loss:', score[0])
    print('Test accuracy:', score[1])
    model.save(args.h5_file)

def main():
    parser = create_parser()
    argcomplete.autocomplete(parser)
    args = parser.parse_args()
    t1 = time.time()
    train(args)
    print("elapsed TIME =",time.time()-t1)

if __name__ == '__main__':
    main()

[Pafrak]

Hi Olivier,
I just tested your code and everything seems of from my side. The code is correctly executing on gvsoc and gapuino (v2). Do you use a gapuino V1 (the one with the USB B connector)? Or did you changed something in other files as well?

[Olivier-Dopeli]

Hi,

I have the Gapuino V2 board and I have the sdk installed on two virtual machines with Ubuntu 16.04 and Ubuntu 18.04 containing the version of the sdk of December and the end of January respectively.

I copied and pasted the example into a folder I created gap_sdk/my_example. And I didn't make any other modifications.

Based on your message, I run the code on gvsoc and I get the same result as mentioned in my first message. What surprises me is that with a single CNN layer model.add (Conv2D (32, kernel_size = (5, 5), input_shape = input_shape)) it doesn't work, but with model.add (Conv2D (64 , kernel_size = (5, 5), input_shape = input_shape)) it works.

The folder of the example I'm using is in attachment.
my_mnist.zip

[Pafrak]

You should use SDK 3.2

https://github.com/GreenWaves-Technologies/gap_sdk/tree/release-v3.2.1

or the master.

Can you test it again?

[Olivier-Dopeli]

I updated the SDK with the following commands:

cd gap_sdk
git checkout master && git pull
git checkout release-v3.2.1
git submodule update --init --recursive
make clean all

In the case of the example I am working on (the folder I sent you) the result remains the same as my first message. In the case of your example nntool/mnist (without any modification on my side), I get the following error :

echo "GENERATING AUTOTILER MODEL"
GENERATING AUTOTILER MODEL
nntool -g -M BUILD_MODEL_8BIT -m mnistModel.c -T BUILD_MODEL_8BIT/tensors  BUILD_MODEL_8BIT/mnist.json
echo "COMPILING AUTOTILER MODEL"
COMPILING AUTOTILER MODEL
gcc -g -o BUILD_MODEL_8BIT/GenTile -I. -I/home/osboxes/gap_sdk/tools/autotiler_v3/include -I/home/osboxes/gap_sdk/tools/autotiler_v3/include -I/home/osboxes/gap_sdk/tools/autotiler_v3/generators/CNN BUILD_MODEL_8BIT/mnistModel.c /home/osboxes/gap_sdk/tools/autotiler_v3/generators/CNN/CNN_Generators.c  /home/osboxes/gap_sdk/tools/autotiler_v3/lib/libtile.a -lSDL2 -lSDL2_ttf
/usr/bin/ld : ne peut trouver -lSDL2_ttf
collect2: error: ld returned 1 exit status
../common/model_rules.mk:86: recipe for target 'BUILD_MODEL_8BIT/GenTile' failed
make: *** [BUILD_MODEL_8BIT/GenTile] Error 1 

Same result by reinstalling the SDK. Procedure followed for the installation :

git clone https://github.com/GreenWaves-Technologies/gap_sdk.git
cd ~/gap_sdk
git checkout release-v3.2.1
git submodule update --init --recursive
source sourceme.sh
make all
make autotiler
make nntool
make openocd

[Pafrak]

On the build error there is a missing dependency:
sudo apt-get install libsdl2-ttf-dev

I'll add it to the readme

[Pafrak]

Moreover I advice you to run it on gvsoc to see if you have any stack overflow, or other memory related errors, with the command:
make run platform=gvsoc

You can have a look at here to have an introduction to gvsoc:

https://greenwaves-technologies.com/gvsoc-the-full-system-simulator-for-profiling-gap-applications/

[Olivier-Dopeli]

Hello @Pafrak ,

Sorry, for the problem of dependency I hadn't taken the time to look.
Anyway, it seems that the problems have been solved.
Thank you very much for your time.

Just one last question, I modified the mnist.c code to get the display of the outputs of the softmax layer via the ResOut variable. I quickly did some different network executions, in most cases the outputs are at 1 for the predicted value and the rest is at 0.
However, I could see values up to 32767 for the following network:

The only modification is one less maxpooling layer between the two conv2D layers.

S1_Conv2d_32x1x5x5_Relu:
S2_Conv2d_64x32x5x5_MaxPool_2x2_Relu:
S3_Linear_10x64x10x10:
S4_SoftMax:

Softmax_Output
0 ; 10922;32767;0;0;32767;0;0;0;0

Do you have any ideas? Or it's just a one-time bad execution.

Thanks again.

[Pafrak]

For the softmax is normal since it is a value between 0 and 1 but quantized in Q1.15 in 16 bits. If you want to convert it back in float you should use a function called FIX2FP (your_value,15)