cbovar / convnetsharp Goto Github PK

I've followed the examples and have trained a MNIST model using the MnistDemo code with serializing the model to json. Now that I have a model I'd like to load the model and test it. I can load/deserialize the model just fine but how to I test it with 28x28 images that I have created separately.

What I'm not clear about is how to get the bitmap data in the format where I can use GetPrediction().

Any tips on how to do this?

Everytime when I run MnistDemo, it throws ArgumentException and then crashes the app. Before cashes, the test accuracy comes up to 85% and then drop back to 4%.

The exception comes from LayerBase.DoForward().

    public virtual Volume<T> DoForward(Volume<T> input, bool isTraining = false)
    {

#if DEBUG
var inputs = input.ToArray();
foreach(var i in inputs)
if (Ops.IsInvalid(i))
throw new ArgumentException("Invalid input!");
#endif
...

Branch: master
IDE: VS2017 and VS2015

Has anybody tested it? I can reproduce it every time.

Features

• CPU and GPU implementations. GPU uses Nvidia Cudnn library
• Generic on numerics (think Net<double> and Net<float>)
• Volumes have now 4 dimensions: width, height, classes, batch size.

Advantages

• Faster. (~100x faster using GTX 760 on forward pass on MNIST vs non-parallel CPU)
• 4 dimensions volumes allows other kinds of layers (e.g. Batch normalization)
• Should be easier to add other volume implementations (mkl ?)

Still to do

• Fluent Net
• Trainers other than vanilla sgd
• Perf improvements
• Serialization
• dotnet standard compability

ConvNetSharp users, do you prefer:

1. Tag current version and commit a different but better ConvNetSharp that will be quite far from a port of ConvNetJS.
   👍 on this comment (:+1:)
2. I leave ConvNetSharp as-is and create another repository for this new version.
   😕 on this comment (:confused:)

Without votes, I will go for option 1, probably this week end

Hello, is there a solution for distributed learning ?
I think the most obvious is to create a small server then send samples to convnetsharp?

Support of RNN is coming soon?

Thank you:)

I created an empty solution, added a new project and added the NuGet-Packages.

Then I added the example code from: https://github.com/cbovar/ConvNetSharp
It works fine with CPU, but as soon as I try to run it GPU it does not find the DLL. If I switch the build to x64 it seems to find the dll, but an exception is thrown:

System.NotImplementedException occurred
HResult=0x80004001
Message=The method or operation is not implemented.
Source=ConvNetSharp.Volume
StackTrace:
at ConvNetSharp.Volume.Single.VolumeBuilder.SameAs(VolumeStorage1 example, Shape shape) at ConvNetSharp.Core.Layers.LayerBase1.DoForward(Volume1 input, Boolean isTraining) at ConvNetSharp.Core.Net1.Forward(Volume`1 input, Boolean isTraining)
at XXX.Program.Main(String[] args) in C:\Users\XXX\documents\visual studio 2017\Projects\XXX\XXX\Program.cs:line 41

on the line:
var prob = net.Forward(x);

Hello,

I am trying to train network such that it will find some object in image. For input I created Volume(someArray,Shape(64,64,1,batchSize)). Images are grayscale with size 64x64. For someArray I created array of size 6464batchSize and it is filled with values of pixels such that every new image begins at position k*4096.

As output layer I used Softmax. Idea was that for every column in image I create separate class. For some reason though, during training I get "Error during calculation". This is because network will eventually adjust its weights, so that all weights coming from convolutional layer are set to very big numbers and then to Infinity.

So I wanted to ask, am I working with volumes right? If I want to train network on multiple images, do I do it in a way, that all images are in one array,one image after another, and network will know how to parse the array according to volume's dimensions? Can someone point me where could I look for errors in my code?

I don't get the concept of 'classes' of the README example.

I know the purpose of this tool is to deal with multidimensional input but I'm a bit stuck

https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Leaky_ReLUs

Hello! My question is: why example uses in train function the shape "new Shape(1, 1, 10, 1)"? Why not "new Shape(10)"?

Hello,

there's a minor bug in AdagradTrainer at line 81. It currently reads:

    vol.SetGradient(i, 0.0); // zero out gradient so that we can begin accumulating anew

But in that point the correct index to use is j, not i:

    vol.SetGradient(j, 0.0); // zero out gradient so that we can begin accumulating anew

Forgive me if I don't submit a pull request, my fork has some more commits on top and branching just for this little fix would be a bit of a hassle.
Thanks again for your great work!

Current, I doing test program MnistDemo.GPU but when I Load file *.json them program error.

Please help me solution.

Thanks.

Hi, first of all thank you for providing wonderful deep learning tools.

I am currently working on some projects that require me to classify image from video, such as face. I have trained the network based on MnistDemo example and saved it to json file so i can use it on main program.

my questions are:

1. how can i set image taken from video as input data?
2. Should i use MnistDemo classes such as circular buffer and dataset?

thank you for your support

I might work on it soon, if you guys don't have time.

This is what I do:

using (var input = new Volume(inputs, new Shape(SolutionStateImageRenderer.ImageWidth, SolutionStateImageRenderer.ImageHeight, 1, minibatch.Length)))
using (var expectedOutput = new Volume(outputs, new Shape(1, 1, classCount, minibatch.Length)))
    this.trainer.Train(input, expectedOutput);

Repeating this multiple times eventually leads to Cuda ErrorOutOfMemory
With some added debug code I can see, that total deallocation in 1 iteration is less than allocation.

Hey,

I'm getting a DllNotFoundException on GpuContext.cs:26.

Additional information: Unable to load DLL 'cudnn64_6.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)

On a fresh Win10 64bit, I'm running VS 2015, VS 2017 and cuda 8. According to the managedCuda Repo, 7.5 should be the matching version, so I installed it in coexistence of version 8, which did not resolve the issue. I did the builds using VS 2015, because the cuda toolkit does not come with a VS 2017 integration yet. Before I start installing and unistalling cuda versions, I'd like to clarify this concern. As the missing dll includes the digit 6, does that mean, that cuda toolkit version 6 is the right version to go with for this repo?

Hi
Thanks for your great works in ConvNetSharp!
I ran your samples and saw that avg. of GPU usage was 15%, how can i use more GPU usage?

Hello @cbovar, thanks for your great work with this porting.

I have a test case where I train a small network, serialize and deserialize it and then compare the predictions of the two networks on some known cases. While this test used to pass with a version of your project pulled on Dec 2, once I updated the library and switched to the "official" serialization method it started giving different results.

Since it is probably a signal of some kind of problem with serialization, I thought the test code may be of help to trace it down. Please find it below:

            Net network = new Net();
            network.AddLayer(new InputLayer(2, 2, 1));

            ConvLayer convLayer = new ConvLayer(2, 2, 2);
            convLayer.Stride = 1;
            convLayer.Pad = 0;
            network.AddLayer(convLayer);

            network.AddLayer(new FullyConnLayer(2));
            network.AddLayer(new ReluLayer());
            network.AddLayer(new FullyConnLayer(1));
            network.AddLayer(new RegressionLayer());

            Volume input = new Volume(2, 2, 1);
            SgdTrainer trainer = new SgdTrainer(network);
            trainer.Momentum = 0.0;

            int epochs = 0;
            double lastError = 0.0;
            bool stop = false;
            do
            {
                // The network is trained to compute higher values
                // when the input contains diagonal patterns
                double error = 0.0;

                // 0 0
                // 0 0 -> 0
                input.Set(0, 0, 0, 0.0);
                input.Set(1, 0, 0, 0.0);
                input.Set(0, 1, 0, 0.0);
                input.Set(1, 1, 0, 0.0);
                trainer.Train(input, 0.0);
                error += trainer.CostLoss;

                // 1 1
                // 0 0 -> 0
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 0.0);
                input.Set(1, 1, 0, 0.0);
                trainer.Train(input, 0.0);
                error += trainer.CostLoss;

                // 0 0
                // 1 1 -> 0
                input.Set(0, 0, 0, 0.0);
                input.Set(1, 0, 0, 0.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 0.0);
                error += trainer.CostLoss;

                // 1 0
                // 1 0 -> 0
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 0.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 0.0);
                trainer.Train(input, 0.0);
                error += trainer.CostLoss;

                // 0 1
                // 0 1 -> 0
                input.Set(0, 0, 0, 0.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 0.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 0.0);
                error += trainer.CostLoss;

                // 0 1
                // 1 0 -> 1
                input.Set(0, 0, 0, 0.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 0.0);
                trainer.Train(input, 1.0);
                error += trainer.CostLoss;

                // 1 0
                // 0 1 -> 1
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 0.0);
                input.Set(0, 1, 0, 0.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 1.0);
                error += trainer.CostLoss;

                // 0 1
                // 1 1 -> 0.5
                input.Set(0, 0, 0, 0.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 0.5);
                error += trainer.CostLoss;

                // 1 1
                // 1 0 -> 0.5
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 0.0);
                trainer.Train(input, 0.5);
                error += trainer.CostLoss;

                // 1 0
                // 1 1 -> 0.5 
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 0.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 0.5);
                error += trainer.CostLoss;

                // 1 1
                // 0 1 -> 0.5
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 0.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 0.5);
                error += trainer.CostLoss;

                // 1 1
                // 1 1 -> 0
                input.Set(0, 0, 0, 1.0);
                input.Set(1, 0, 0, 1.0);
                input.Set(0, 1, 0, 1.0);
                input.Set(1, 1, 0, 1.0);
                trainer.Train(input, 0.0);
                error += trainer.CostLoss;

                error /= 12.0;
                epochs++;

                if (error < 0.001)
                {
                    stop = true;
                }
                else if (epochs > 1)
                {
                    // Stop if the error does not improve sufficiently
                    double gain = (lastError - error) / lastError;
                    stop = (gain < 0.1);
                }

                lastError = error;

            } while (!stop);

            string serialized = network.ToJSON();
            Net network2 = SerializationExtensions.FromJSON(serialized);

            // We now test that the two networks (original and deserialized)
            // compute the same results: we don't care if the prediction
            // is correct, just that the two networks predict the same thing
            double output = 0.0, output2 = 0.0;

            // 0 0
            // 0 0
            input.Set(0, 0, 0, 0.0);
            input.Set(1, 0, 0, 0.0);
            input.Set(0, 1, 0, 0.0);
            input.Set(1, 1, 0, 0.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 1
            // 0 0
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 0.0);
            input.Set(1, 1, 0, 0.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 0 0
            // 1 1
            input.Set(0, 0, 0, 0.0);
            input.Set(1, 0, 0, 0.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 0
            // 1 0
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 0.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 0.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 0 1
            // 0 1
            input.Set(0, 0, 0, 0.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 0.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 0 1
            // 1 0
            input.Set(0, 0, 0, 0.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 0.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 0
            // 0 1
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 0.0);
            input.Set(0, 1, 0, 0.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 0 1
            // 1 1
            input.Set(0, 0, 0, 0.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 1
            // 1 0
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 0.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 0
            // 1 1
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 0.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 1
            // 0 1
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 0.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

            // 1 1
            // 1 1
            input.Set(0, 0, 0, 1.0);
            input.Set(1, 0, 0, 1.0);
            input.Set(0, 1, 0, 1.0);
            input.Set(1, 1, 0, 1.0);
            output = network.Forward(input).Get(0, 0, 0);
            output2 = network2.Forward(input).Get(0, 0, 0);
            Assert.AreEqual(output, output2, 0.001);

Great library, only one in C#!!

I was trying to load a torch model but it was impossible to set up weights and biases to see if ConvNetSharp has the same data order. Any efforts on this side? How can I test if the model I build with V will be the same as Torch, at least for the basic layers; SpatialConvolutionMM, Relus, Linea, etc.

Cudafy doesn't seem to be maintained anymore and is not compatible with latest CUDA

What's a volume ? What's a shape ?

Hello,

System.MissingMethodException: ''ConvNetSharp.Core.Layers.Single.ConvLayer'

After 27.3.17 update, I can't build any examples - project's files ConvNetSharp.Core and ConvNetSharp.Volume couldn't be loaded.

Error list:

1. There is no target object "GetTargetPath" at file ConvNetSharp.Volume.csproj
2. Unrecognized "Version" attribute in the at file ConvNetSharp.Core.csproj

Are there any plans to add simple RNN layers?
V(tanh(Ux + Ws))

Hi, thanks again for the help for the other day

is there any way to display confidence level of each label when doing classification?

for example:

Pict X is a dog, confidence level: 90%
Pict X is a human, confidence level: 100%

thanks for the support :)

Seems like an awesome project. Can you make it available on NuGet?

I use code after:
Save: ConvNetSharp.Core.Serialization.SerializationExtensions.ToJson(_net)
Load: ConvNetSharp.Core.Serialization.SerializationExtensions.FromJson(Link+"Data.json")

data.txt

Save is OK but when I load file to use them it display Error, Please help me.
Thanks.

Are you thinking in adding Inception Layers?

Using GPU MNIST demo (from GPU branch) and CPU MNIST demo

• 50k iterations
• SGD trainer (BatchSize = 20, L2Decay = 0.001, LearningRate = 0.01)
• Same random seed

GPU implem is slower than CPU implem iteration-wise but computes faster:

By dumping weights and gradients, it seems this issue is linked to rounding errors, mainly from the backward pass.

Small differences between GPU and CPU are normal because:

• operation order may be different
• .Net may use higher precision than the result presicion (https://msdn.microsoft.com/en-us/library/aa691146(v=vs.71).aspx)

Same phenomenon happens using single precision as well. Screenshot shows only one try but the error is consistent over many tries/seeds.

This is troubling because GPU loss never seems to converge at the CPU level. GPU implemis pretty much worthless as is.

Hello!

Is it possible to create a segmentation network using your library? As I know, for that I need an unpooling and deconvolutional layers.

MNIST data should be automatically downloaded if not already present.

I'm looking to make a CNN which concatenates new information in on the first fully connected layer. Am I missing something already included or can you implement this feature? Graph flow would go:

Image
vvvvvv
Convolution Layer
vvvvvv
Fully Connected Layer <<< Additional Information
vvvvvv
Output Layer

Net.AddLayer() should throw if an inadequate layer is added but should not automatically create layers.

RL has been added to original ConvNetJS.
Will you be adding that too?
Any plans for LSTM?

Thanks

GPU
Memory leaks in AdamTrainer with RegressionLayer
Possible reason method TrainImplem();

public virtual void Train(Volume<T> x, Volume<T> y)
        {   Forward(x);
            Backward(y);
            var batchSize = x.Shape.GetDimension(3);
            var chrono = Stopwatch.StartNew();
            TrainImplem();
            this.UpdateWeightsTimeMs = chrono.Elapsed.TotalMilliseconds/batchSize;
        }

Flip modes: LeftRight. UpDown and Both

I tried importing ConvNetSharp in unity but it kept throwing errors, has anyone used this library in unity?

Thank you for this great library,

I'm trying to use the pure flow approach, but I have not been able to understand how to do it looking at ExampleCpuSingle....
Could you please provide a sample code showing how to build and train the net in the example image ?

Thank you very much

E.g.

var net = FluentNet.Create(24, 24, 1)
                   .Conv(5, 5, 8).Stride(1).Pad(2)
                   .Relu()
                   .Pool(2, 2).Stride(2)
                   .Conv(5, 5, 16).Stride(1).Pad(2)
                   .Relu()
                   .Pool(3, 3).Stride(3)
                   .FullyConn(10)
                   .Softmax(10)
                   .Build();

Possibility to merge network branches:

var branch1 = FluentNet.Create(10, 10, 2)
                .Relu()
                .FullyConn(10);
var branch2 = FluentNet.Create(10, 10, 2)
                .Relu()
                .FullyConn(20);

var net = FluentNet.Merge(branch1, branch2)
                .FullyConn(5)
                .Softmax(5)
                .Build();

net.Forward(new[] {new Volume(10, 10, 2), new Volume(10, 10, 2) }); // 2 inputs

The WeightGradients is a private double in Volume class but no definition in IVolumne. This will result in error in class FullyConnLayer when you use.

var temp = volume.WeightGradients;

Hello,
What's about dropout layer?
I want to test ConvNetSharp on CIFAR-10 and I need dropout layer. Please add it.

This prevents custom kernels to be loaded and throws an exception at start when using GPU

The example in README.md refers to prob.Weights[0]. However, as per this commit the .Weights field in Volume is now private and must be accessed via .Get(). Really minor change, just kind of annoying since the provided example doesn't actually compile and forces one to dig through the commit history to figure out how to fix it!

Dear,

I have a problem to save net and reload net when i training finished. Can you help me what to do??

Thanks!

Hey convent users, Happy new year !!!

excuse me, is there any way to store trained network into file file trained-model.file so we can use that trained model in separate program? thanks for the response