Comments (15)
- It seems that
size
can be computed outside the for loop - I understand your training accuracy doesn't get better when you provide more training data. Have you tried with a simpler network ? I'm not sure I understand the way you compute the kernel size of convolution layers
- If you have a full code source that I could run, that would be easier for me to help you
- The training algorithms (Sgd and Adam) are inspired by original implementation of ConvNetJS. You could look at https://arxiv.org/pdf/1609.04747.pdf
from convnetsharp.
I have an array of doubles that comes from processed features of subjects. right now im trying to run this code but the training accuracy is off.
SgdTrainer Tr = new SgdTrainer(net)
{
LearningRate = 0.01,
BatchSize = 500,
L2Decay = 0.001,
Momentum = 0.9
};
net.AddLayer(new InputLayer(28, 28, 1));
net.AddLayer(new ConvLayer(5, 5, 8) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(2, 2) { Stride = 2 });
net.AddLayer(new ConvLayer(5, 5, 16) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(3, 3) { Stride = 3 });
net.AddLayer(new FullyConnLayer(10));
net.AddLayer(new SoftmaxLayer(10));
double[] d = new double[12 * 63];
for (int k = 0; k < 10; k++)
{
int count = 0;
for (int i = 0; i < 12; i++)
{
for (int j = 0; j < UserDATA[k].GetLength(1); j++)
{
d[count] = UserDATA[k][i, j];
count++;
}
}
var x = BuilderInstance.Volume.From(d, new Shape(12, 63, 1));
double[] z = new double[10];
for (int t = 0; t < z.Length; t++)
{
z[t] = 0.0;
}
z[k] = 1.0;
var zx = BuilderInstance.Volume.From(z, new Shape(1, 1, 10, 1));
for (int g = 0; g < Convert.ToInt32(NumberOfTrainingSteps.Text); g++)
{
Tr.Train(x, zx); // train the network, specifying that x is class zero
}
}
double[] ts = new double[12 * 63];
double[] testd = new double[12 * 63];
for (int k = 0; k < 10; k++)
{
int count = 0;
for (int i = 0; i < 12; i++)
{
for (int j = 0; j < UserDATA[k].GetLength(1); j++)
{
testd[count] = UserDATA[k][i, j];
if (k == 0)
ts[count] = UserDATA[k][i, j];
count++;
}
}
var x = BuilderInstance.Volume.From(testd, new Shape(12, 63, 1));
var prob = net.Forward(x);
TestCON.Text += "\r\n" + " " + k + " " + prob.Get(k);
TestCON.Text += "\r\n" + k + " cl 0 prob " + prob.Get(0);
}
it seems that NumberOfTrainingSteps does not give me any increase in accuracy. but that can be expected because im not feeding any new data to the network. Thing is, even if i do train it on other examples nothing changes. Also, what is the BatchSize in trainer responsible for? Also, as i understand the input layer size should correspond with the amount of data points i feed to the network i.e. 28x28x1 should take no more than 784 data points?
from convnetsharp.
- You should present a different input every time you call
Train
method. It seems you call NumberOfTrainingSteps timesTrain
with the same data. This will make the network forget previous data in the dataset. - BatchSize is used in the trainers to normalize the gradients. It seems the BatchSize information is duplicated: in the trainer and in the 4th input volume dimension. I think it is possible to get rid of the one on the trainer but haven't done it yet (it's some relics from original ConvNetJS implem)
- You should feed the network during training and inference with volume of shape 28x28x1xBatchSize. With BatchSize = 1 it should take 784 data points exactly (it seems you feed less data than that)
from convnetsharp.
Can you tell me what learningRate, L2Decay, Momentum represent? Also, is it possible to use the same data samples to train the network? Do you have functions that mutate the weights(simulated annealing, freezing, evolution multidimensional optimisation) or functions that separate epochs in training of the network? Also, if i use different trainers to train the network on the same data samples will it change anything performance wise?
from convnetsharp.
The learning rate determines the size of the steps we take to reach a (local) minimum. Basically the gradients are multiplied by the learning rate before being used to update the parameters to optimize. (see here in the code)
L1Decay and L2Decay are supposed to be used for regularization. You've made me realize that I still haven't implemented them. So these parameters are useless. I will get rid of them in the meanwhile.
Momentum is a method that helps accelerate SGD. You can look at section 4.1 of https://arxiv.org/pdf/1609.04747.pdf. (see here in the code)
The functions that mutate weights are called Trainers in ConvNetSharp. SgdTrainer / AdamTrainer for ConvNetSharp.Core and SgdTrainer / AdamTrainer for ConverNetSharp.Flow
Using different trainers will impact the performance of the network: Some training algorithms are more adapted to some kind of tasks.
I am not sure I understand "functions that separate epochs in training of the network". If it's a function to split data set in training / testing / validating, there is no such function in this library.
from convnetsharp.
Thank you for information! I have another question: im testing network after every training step and having problems with probability output. 100 different test samples outputs the same probability. As i understand the output should be different with every new test sample. What may cause that? It seems to me that network forgets previous training data or i simply can't see errors in my code. Here is the code: Net net = new Net();
AdamTrainer Trex = new AdamTrainer(net)
{
LearningRate = LR,
BatchSize=1
};
net.AddLayer(new InputLayer(999, 705, 1));
net.AddLayer(new ConvLayer(11, 11, 5) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(2, 2) { Stride = 2 });
net.AddLayer(new ConvLayer(5, 5, 16) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(2, 2) { Stride = 2 });
net.AddLayer(new ConvLayer(3, 3, 20) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(2, 2) { Stride = 2 });
net.AddLayer(new ConvLayer(2, 2, 30) { Stride = 1, Pad = 1 });
net.AddLayer(new ReluLayer());
net.AddLayer(new FullyConnLayer(2));
net.AddLayer(new SoftmaxLayer(2));
for (int yh = 0; yh < 100; yh++)
{
var x = BuilderInstance.Volume.From(TrueSamp[yh], new Shape(999, 705, 1));
var y = BuilderInstance.Volume.From(FalseSamp[yh], new Shape(999, 705, 1));
var zx = BuilderInstance.Volume.From(new[] { 1.0, 0.0 }, new Shape(1, 1, 2, 1));
var zy = BuilderInstance.Volume.From(new[] { 0.0, 1.0 }, new Shape(1, 1, 2, 1));
Trex.Train(x, zx); // train the network, specifying that x is class 0
avloss += Trex.Loss;
loss += "\r\n" + Trex.Loss;
Trex.Train(y, zy); // train the network, specifying that y is class 1
avloss += Trex.Loss;
loss += "\r\n" + Trex.Loss;
Random rand = new Random();
double[] truesamp = new double[shit];
for (int i = 0; i < 100; i++)
{
truesamp = TrueSampTest[rand.Next(0, 100)];
var rq = BuilderInstance.Volume.From(truesamp, new Shape(999 * 705, 1, 1));
var probq = net.Forward(rq);
trueoutput += "\r\n" + probq.Get(0);
}
trueoutput += "yh ="+yh ;
for (int i = 0; i < 100; i++)
{
double[] falsesamples = new double[shit];
falsesamples = FalseSampTest[rand.Next(0, 100)];
var rx = BuilderInstance.Volume.From(falsesamples, new Shape(999 * 705, 1, 1));
var proby = net.Forward(rx);
falseoutput += "\r\n" + +proby.Get(0);
}
falseoutput += "yh =" + yh;
}
from convnetsharp.
Does the loss decrease?
It should output the same proba when yh is low but should not when yh start to grow.
from convnetsharp.
loss decreases as it should. According to the paper you сited.
from convnetsharp.
The input shape you use for testing seems odd: new Shape (999 * 705, 1, 1) instead of new shape(999, 705, 1). I'm not sure that's the source of the problem but it'd be interesting to fix that.
from convnetsharp.
Also, could you try decreasing the learning rate and post a new plot the loss? Maybe divide it by 10.
from convnetsharp.
I've changed the shape in testing method. No changes in proba occurred. Also tried decreasing learning rate, here is the plot.
from convnetsharp.
What is the value of LR?
Any chance to have the full code so I can run it ? I think I just need FalseSamp, TrueSamp, FalseSampTest, TrueSampTest
from convnetsharp.
Right now LR is 0.001. Here is the main code. Im reading data from files like the one i attached. Each line in file is a set of coordinates with corresponding value. One file is one training sample. I've changed architecture of layers a bit as it gives slightly better results.
Net net = new Net();
string[] arr = Directory.GetFiles("C:\Users\USER\Desktop\Norm_podp\YST", ".");
string[] arrTest = Directory.GetFiles("C:\Users\USER\Desktop\BaS_PFS\NORM", ".");
string[] arrSig = Directory.GetFiles("C:\Users\USER\Desktop\Norm_podp\NORM", ".");
double[,,] parce = new double[200, 1000, 1000];
double[,,] parceTest = new double[100, 1000, 1000];
double[,,] parceSig = new double[200, 1000, 1000];
double[][] TrueSamp = new double[200][];
double[][] TrueSampTEst = new double[100][];
double[][] FalseSamp = new double[200][];
int dimentionSize = (999 * 705);
for (int k = 0; k < 200; k++)
{
for (int i = 0; i < 1000; i++)
{
for (int j = 0; j < 1000; j++)
{
parce[k, i, j] = -1;
if (k < 100)
parceTest[k, i, j] = -1;
parceSig[k, i, j] = -1;
}
}
}
int count = 0;
foreach (string file in arr)
{
string[] text = System.IO.File.ReadAllLines(file);
for (int i = 0; i < text.Length; i++)
{
string[] sp = text[i].Split(';');
int x = Convert.ToInt32(sp[0]);
int y = Convert.ToInt32(sp[1]);
parce[count, x, y] = Convert.ToInt32(sp[2]);
}
count++;
}
count = 0;
foreach (string file in arrTest)
{
string[] text = System.IO.File.ReadAllLines(file);
for (int i = 0; i < text.Length; i++)
{
string[] sp = text[i].Split(';');
int x = Convert.ToInt32(sp[0]);
int y = Convert.ToInt32(sp[1]);
parceTest[count, x, y] = Convert.ToInt32(sp[2]);
}
count++;
}
count = 0;
foreach (string file in arrSig)
{
string[] text = System.IO.File.ReadAllLines(file);
for (int i = 0; i < text.Length; i++)
{
string[] sp = text[i].Split(';');
int x = Convert.ToInt32(sp[0]);
int y = Convert.ToInt32(sp[1]);
parceSig[count, x, y] = Convert.ToInt32(sp[2]);
}
count++;
}
TestCON.Text += "Files loaded \r\n";
count = 0;
int numberoftrainingex = 200;
double LR = 0.001;
string trueoutput = "";
string trueoutputfalsesamp = "";
string falseoutput = "";
string falseoutputfalsesamp = "";
int com = 0;
for (int k = 0; k < numberoftrainingex; k++)
{
TrueSamp[k] = new double[shit];
if(k<100)
TrueSampTEst[k] = new double[shit];
FalseSamp[k] = new double[shit];
}
for (int k = 0; k < numberoftrainingex; k++)
{
for (int i = 0; i < 999; i++)
{
for (int j = 0; j < 705; j++)
{
TrueSamp[k][com] = parce[k, i, j];
if (k < 100)
TrueSampTEst[k][com]= parceTest[k, i, j];
FalseSamp[k][com] = parceSig[k, i, j];
com++;
}
}
com = 0;
}
var avloss = 0.0;
string loss="";
Net<double> net = new Net<double>();
AdamTrainer Trex = new AdamTrainer(net)
{
LearningRate = LR,
BatchSize=1
};
net.AddLayer(new InputLayer(999, 705, 1));
net.AddLayer(new ConvLayer(11, 11, 5) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(3, 3) { Stride = 2 });
net.AddLayer(new ConvLayer(5, 5, 16) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(3, 3) { Stride = 2 });
net.AddLayer(new ConvLayer(3, 3, 40) { Stride = 1, Pad = 2 });
net.AddLayer(new ReluLayer());
net.AddLayer(new PoolLayer(3, 3) { Stride = 2 });
net.AddLayer(new ConvLayer(3, 3, 80) { Stride = 1, Pad = 1 });
net.AddLayer(new FullyConnLayer(2));
net.AddLayer(new SoftmaxLayer(2));
for (int yh = 0; yh < 100; yh++)
{
var x = BuilderInstance.Volume.From(TrueSamp[yh], new Shape(999, 705));
var y = BuilderInstance.Volume.From(FalseSamp[yh], new Shape(999, 705));
var zx = BuilderInstance.Volume.From(new[] { 1.0, 0.0 }, new Shape(1, 1, 2, 1));
var zy = BuilderInstance.Volume.From(new[] { 0.0, 1.0 }, new Shape(1, 1, 2, 1));
Trex.Train(x, zx); // train the network, specifying that x is class zero
avloss += Trex.Loss;
loss += "\r\n" + Trex.Loss;
Trex.Train(y, zy); // train the network, specifying that x is class zero
avloss += Trex.Loss;
loss += "\r\n" + Trex.Loss;
Random rand = new Random();
double[] truesamp = new double[shit];
truesamp = TrueSamp[yh];
var rq = BuilderInstance.Volume.From(truesamp, new Shape(999 * 705));
var probq = net.Forward(rq);
trueoutput += "\r\n" + probq.Get(0);
trueoutputfalsesamp += "\r\n" + probq.Get(1);
double[] falsesamples = new double[shit];
falsesamples = FalseSamp[yh];
var rx = BuilderInstance.Volume.From(falsesamples, new Shape(999 * 705));
var proby = net.Forward(rx);
falseoutputfalsesamp += "\r\n" +proby.Get(0);
falseoutput += "\r\n" +proby.Get(1);
}
avloss= avloss/200;
TestCON.Text += "av loss" + avloss;
TestCON.Text += "loss by steps" + "\r\n" + loss;
TestCON.Text += " test samples of class1 beeing class1" + "\r\n";
TestCON.Text += trueoutput;
TestCON.Text += " test samples of class1 beeing class2" + "\r\n";
TestCON.Text += trueoutputfalsesamp;
TestCON.Text += " test samples of class2 beeing class1" + "\r\n";
TestCON.Text += falseoutput;
TestCON.Text += "test samples of class2 beeing class2" + "\r\n";
TestCON.Text += falseoutputfalsesamp;
com = 0;
TestCON.Text +="TEST 1";
for (int j = 0; j < 100; j++)
{
Random rand = new Random();
double[] truesamp = new double[shit];
truesamp = TrueSamp[j+100];
var rq = BuilderInstance.Volume.From(truesamp, new Shape(999 * 705));
var probq = net.Forward(rq);
TestCON.Text += "\r\n" + probq.Get(0);
}
TestCON.Text += "TEST 1 1";
for (int j = 0; j < 100; j++)
{
Random rand = new Random();
double[] truesamp = new double[shit];
truesamp = TrueSamp[j+100];
var rq = BuilderInstance.Volume.From(truesamp, new Shape(999 * 705));
var probq = net.Forward(rq);
TestCON.Text += "\r\n" + probq.Get(1);
}
TestCON.Text += "TEST 2";
for (int j = 0; j < 100; j++)
{
Random rand = new Random();
double[] truesamp = new double[shit];
truesamp = TrueSampTEst[j];
var rq = BuilderInstance.Volume.From(truesamp, new Shape(999 * 705));
var probq = net.Forward(rq);
TestCON.Text += "\r\n" + probq.Get(0);
}
TestCON.Text += "TEST 2 2";
for (int j = 0; j < 100; j++)
{
Random rand = new Random();
double[] truesamp = new double[shit];
truesamp = TrueSampTEst[j];
var rq = BuilderInstance.Volume.From(truesamp, new Shape(999 * 705));
var probq = net.Forward(rq);
TestCON.Text += "\r\n" + probq.Get(1);
}
from convnetsharp.
Here is the loss plot and probability plot after each epoch
from convnetsharp.
I had simillar problems and net started working when I normalized input (0 - 1)
from convnetsharp.
Related Issues (20)
- Wrong gradient computation in Flow when batch size > 1
- How to load FluentNet<T> from json file? HOT 8
- Executable file locks after a crash. HOT 3
- using Dropout HOT 2
- Creating convolutional network? HOT 1
- how to concat multi layers? HOT 3
- ConvNetSharp compare to other Deep Learning frameworks HOT 9
- TrainerBase class is not updated in NuGet HOT 9
- How to make deconvolution neural network U-Net. HOT 1
- Fluent serialization not working
- [Question] Network with different output kind (softmax & scalar) HOT 5
- how to create 1d Convolutional network with pading? HOT 3
- Import ONNX models / models from other frameworks HOT 2
- "disposable" not defined HOT 5
- "Invalid Input!" occurs at fully connected layer during training HOT 3
- Getting started - basic network won't classify two distinct patterns HOT 7
- Question HOT 2
- How to implement the following layer?
- Show Graph is not work
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from convnetsharp.