amten / neuralnetwork

Java (convolutional or fully-connected) neural network implementation with plugin for Weka. Uses dropout and rectified linear units.

What's this?

Java (convolutional or fully-connected) neural network implementation with plugin for [Weka] (http://www.cs.waikato.ac.nz/ml/weka/). Uses dropout and rectified linear units. Implementation is multithreaded and uses [MTJ] (https://github.com/fommil/matrix-toolkits-java) matrix library with native libs for performance.

Installation

Weka

Go to https://github.com/amten/NeuralNetwork/releases/latest to find the latest release. Download the files NeuralNetwork.zip and BLAS-dlls.zip. In Weka, go to Tools/Package Manager and press the "File/URL" button. Browse to the NeuralNetwork.zip file and press "ok".

Important! For optimal performance, you need to install native matrix library files.
Windows: Unzip the BLAS-dlls.zip file to Wekas install dir (".../Program Files/Weka-3-7").
Ubuntu: Run "sudo apt-get install libatlas3-base libopenblas-base" in a terminal window.

Standalone

This package was made mainly to be used from the Weka UI, but it can be used in your own java code as well.

Go to https://github.com/amten/NeuralNetwork/releases/latest to find the latest release. Download the file NeuralNetwork.zip and unzip.

Include the files NeuralNetwork.jar, lib/mtj-1.0-snapshot.jar, lib/opencsv-2.3.jar in your classpath.

Important! For optimal performance, you need to install native matrix library files.
Windows: Unzip the BLAS-dlls.zip file to the directory where you execute your application, or any other directory in the PATH.
Ubuntu: Run "sudo apt-get install libatlas3-base libopenblas-base" in a terminal window.

Usage

Weka

In Weka, you will find the classifier under classifiers/functions/NeuralNetwork. For explanations of the settings, click the "more" button.

Note 1: If you start Weka with console (alternative available in the windows start menu), you will get printouts of cost during each iteration of training and you can press enter in the console window to halt the training.

Note 2: When using dropout as regularization, it might still be a good idea to keep a small weight penalty. This keeps weights from exploding and causing overflows.

Note 3: When using convolutional layers, it seems to be most efficient to use batch-size=1 (i.e. Stochastic Gradient Descent)

Standalone

Example code showing classification and regression can be found here: https://github.com/amten/NeuralNetwork/tree/master/src/amten/ml/examples

License

Free to copy and modify. Please include author name if you copy code.

neuralnetwork's Issues

saving and retrieving trained NN

Hello,
thank you for writing such a good code, it's really easy to get it running and experimenting quickly!
I'm using the NNs to classify a data set unrelated to weka, therefore I need to train my NN before using it and then use the trained NN with data from the users, in a live fashion. Since training takes several hours, I can't afford to train it every time I need to use it. I haven't found any methods in your code to save the NN's weights, nor configuration. I found in a weka related page the following code:

Debug.saveToFile(filename+"_nn.dat", nn);
then, to load in the classifying part:
SerializedClassifier classifier = new SerializedClassifier();
classifier.setModelFile(new File("/path/to/weka-neural-network"));

Will this work with your code,i.e. if I use instead of "SerializedClassifier"
NNParams params = new NNParams();
params.numClasses = 10; // 10 digits to classify
params.hiddenLayerParams = myNNconfig
and then
classifier.setModelFile(new File("/path/to/weka-neural-network"));?

Thank you very much for your help!

Invoking NeuralNetwork from command line

Hi there,

First of all, thanks for writing this! I'm currently trying this out for the Kaggle digits competition. I was particular excited by the results you managed to get.

I'm having trouble invoking this algorithm from command-line however, and I think that may have to do with the fact that your algorithm's flag for the number of threads is -t, whereas this is what Weka uses to denote the training file.

For example:

java -Xmx6500M -cp $WEKA_LATEST weka.Run weka.classifiers.functions.NeuralNetwork -lr 0.0 -wp 1.0E-8 -i 250 -bs 1 -t 4 -hl 20-5-5-2-2,100-5-5-2-2 -di 0.2 -dh 0.5 -iw 0 -t trainfile.arff -no-cv -d modelfile.model

As you can see, -t is also used to denote the training .arff file.

Removing -t 4 won't fix the problem, either.

Anyways a fix for this would be awesome, since you can end up saving quite a bit of memory using Weka from the command-line (as you don't have to load the entire dataset into memory which is what happens with the GUI).

Perhaps there's a workaround for this that won't involve any changing of the code, but I don't know of anything like this yet. :)

Cheers

I can't save the params of trained network

Hi, thanks for writing such a good code, but i cant save the weights of network. I used the method described in other issue #12 , but it didn't worked out for me. Here is the code

public void loadNN(String filename){

    mySoftmax = myParams.numClasses > 1;
    myNumOutputs = myParams.numClasses;

    // Add null params for layer 0, which is just the input and last layer, which is just output.
    myLayerParams = new NNParams.NNLayerParams[myParams.hiddenLayerParams.length+2];
    System.arraycopy(myParams.hiddenLayerParams, 0, myLayerParams, 1, myParams.hiddenLayerParams.length);
    myLayerParams[myLayerParams.length-1] = new NNParams.NNLayerParams(myNumOutputs);
    myParams.numThreads = myParams.numThreads > 0 ? myParams.numThreads : Runtime.getRuntime().availableProcessors();
    myExecutorService = Executors.newFixedThreadPool(myParams.numThreads);

    int numLayers = myLayerParams.length;

    initThetas();

    for (int layer = 1; layer < numLayers; layer++)
        try{
            myThetas[layer]=MatrixUtils.readCSV(filename+"_l"+layer+".csv",',', 0);
        }catch(Exception e) {System.out.println(e.toString());}
}

public void saveNN(String filename){
    int numLayers = myLayerParams.length;

    for (int layer = numLayers-1; layer > 0; layer--)
        try{
            int l = numLayers-layer-1;
            MatrixUtils.writeCSV(this.myThetas[layer],filename+"_l"+ layer +".csv");
        }catch(Exception e) {System.out.println(e.toString());}
}

Thanks in advance for your reply.

1D-Convolution

Hello,
I'm using your tool for speech processing (emotion recognition). So I have only 1D-data, consisting of labelled raw audio wave values. I tried to use CNNs for recognition (using the Weka Gui), but I get an index-out-of-bounds-exception, stating that "Matrix size cannot be negative". My values for the hidden layers are: 20-5-1-2-2, 20-5-1-2-2.
Can you help me?
Thanks!

Seed parameter

I'm trying to use NeuralNetwork for regression. However, every researches with same configration output different results. Which is bad for research.

Can I request seed parameter for the same result?

About Best NN struct for predict Car price example ?

Could you tell me which is the best NN struct for run predict Car price example?

Thank in advance

Unsupported major.minor version 51.0

Hi there,

Thanks in regards to the previous bug-fix. I've hit another roadblock however, when I install the package and try and run Weka on Mac, I get this:

Exception in thread "main" java.lang.UnsupportedClassVersionError: weka/classifiers/functions/NeuralNetwork : Unsupported major.minor version 51.0
    java.lang.ClassLoader.defineClass1(Native Method)
    java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
    java.lang.ClassLoader.defineClass(ClassLoader.java:621)
    java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    java.security.AccessController.doPrivileged(Native Method)
    java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    java.lang.Class.forName0(Native Method)
    java.lang.Class.forName(Class.java:171)
    weka.core.ClassDiscovery.find(ClassDiscovery.java:343)
    weka.gui.GenericPropertiesCreator.generateOutputProperties(GenericPropertiesCreator.java:528)
    weka.gui.GenericPropertiesCreator.execute(GenericPropertiesCreator.java:620)
    weka.gui.GenericPropertiesCreator.execute(GenericPropertiesCreator.java:596)
    weka.core.converters.ConverterUtils.initialize(ConverterUtils.java:730)
    weka.core.converters.ConverterUtils.<clinit>(ConverterUtils.java:711)
    java.lang.Class.forName0(Native Method)
    java.lang.Class.forName(Class.java:171)
    weka.core.ClassDiscovery.find(ClassDiscovery.java:343)
    weka.gui.GenericPropertiesCreator.generateOutputProperties(GenericPropertiesCreator.java:528)
    weka.gui.GenericPropertiesCreator.execute(GenericPropertiesCreator.java:620)
    weka.gui.GenericPropertiesCreator.<clinit>(GenericPropertiesCreator.java:162)
    weka.core.WekaPackageManager.refreshGOEProperties(WekaPackageManager.java:895)
    weka.core.WekaPackageManager.loadPackages(WekaPackageManager.java:889)
    weka.core.WekaPackageManager.loadPackages(WekaPackageManager.java:817)
    weka.gui.GenericObjectEditor.determineClasses(GenericObjectEditor.java:174)
    weka.gui.GenericObjectEditor.<clinit>(GenericObjectEditor.java:248)
    weka.gui.GUIChooser.<init>(GUIChooser.java:707)
    weka.gui.GUIChooser.createSingleton(GUIChooser.java:259)
    weka.gui.GUIChooser.main(GUIChooser.java:1558)

    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:171)
    at weka.core.ClassDiscovery.find(ClassDiscovery.java:343)
    at weka.gui.GenericPropertiesCreator.generateOutputProperties(GenericPropertiesCreator.java:528)
    at weka.gui.GenericPropertiesCreator.execute(GenericPropertiesCreator.java:620)
    at weka.gui.GenericPropertiesCreator.execute(GenericPropertiesCreator.java:596)
    at weka.core.converters.ConverterUtils.initialize(ConverterUtils.java:730)
    at weka.core.converters.ConverterUtils.<clinit>(ConverterUtils.java:711)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:171)
    at weka.core.ClassDiscovery.find(ClassDiscovery.java:343)
    at weka.gui.GenericPropertiesCreator.generateOutputProperties(GenericPropertiesCreator.java:528)
    at weka.gui.GenericPropertiesCreator.execute(GenericPropertiesCreator.java:620)
    at weka.gui.GenericPropertiesCreator.<clinit>(GenericPropertiesCreator.java:162)
    at weka.core.WekaPackageManager.refreshGOEProperties(WekaPackageManager.java:895)
    at weka.core.WekaPackageManager.loadPackages(WekaPackageManager.java:889)
    at weka.core.WekaPackageManager.loadPackages(WekaPackageManager.java:817)
    at weka.gui.GenericObjectEditor.determineClasses(GenericObjectEditor.java:174)
    at weka.gui.GenericObjectEditor.<clinit>(GenericObjectEditor.java:248)
    at weka.gui.GUIChooser.<init>(GUIChooser.java:707)
    at weka.gui.GUIChooser.createSingleton(GUIChooser.java:259)
    at weka.gui.GUIChooser.main(GUIChooser.java:1558)

The weka.jar I use is the exact same one I use on Windows, and there are no issues on Windows.

I'm assuming I'll have to recompile your Java files on Mac and Linux to get it working? Let me know what you think.

EDIT - I checked the Java version on the machines that worked (and the ones that didn't as well) and I think it might be related to the version. It seems to crap out on 1.6 but it works on 1.7. If this is the case, then I'll have to either re-compile your code using JDK 1.7 or I'll have to add Java 1.7 to my classpath.

EDIT2 - yes using Java 1.7 solved the issue. Feel free to close this ticket, or whatever. :)

Cheers :)

weka version

In new version weka 3.6.11, I did not find the Tools/package manage , How can I add the package?

Test Neural Network

Hi, I am using your soruce code of neural network in https://github.com/amten/NeuralNetwork
I run the classification example NNClassificationExample.java but I dont know how can I test the neural network after the training phase.

I try with predictedClasses = nn.getPredictedClasses(xCV); and get the next output:

0.999999966203474 1.8422446731342536E-14 2.8438987740204348E-8
1.6397789704005972E-9 0.9999945945378662 1.329060718270638E-7
5.901015821081087E-9 0.9999846853657705 1.087386918868424E-6

How can I classify after train the neural network?

please help when i install the package on weka this error occur

java.lang.NullPointerException
at weka.core.WekaPackageLibIsolatingClassLoader.checkForNativeLibs(WekaPackageLibIsolatingClassLoader.java:220)
at weka.core.WekaPackageLibIsolatingClassLoader.init(WekaPackageLibIsolatingClassLoader.java:145)
at weka.core.WekaPackageLibIsolatingClassLoader.(WekaPackageLibIsolatingClassLoader.java:126)
at weka.core.WekaPackageClassLoaderManager.addPackageToClassLoader(WekaPackageClassLoaderManager.java:369)
at weka.core.WekaPackageManager.initializeAndLoadUnofficialPackage(WekaPackageManager.java:2285)
at weka.core.WekaPackageManager.installPackageFromArchive(WekaPackageManager.java:2272)
at weka.gui.PackageManager$UnofficialInstallTask.doInBackground(PackageManager.java:779)
at weka.gui.PackageManager$UnofficialInstallTask.doInBackground(PackageManager.java:723)
at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at javax.swing.SwingWorker.run(SwingWorker.java:334)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[error launching weka] java.lang.UnsupportedClassVersionError: weka/gui/GUIChooser : Unsupported major.minor version 52.0

Hello,
I'm using weka 3.6.10-2. After installation, that's the error I have when I tried to launch weka:
$weka
[warning] /usr/bin/weka: Unable to locate libsvm.jar in /usr/share/java
Exception in thread "main" java.lang.UnsupportedClassVersionError: weka/gui/GUIChooser : Unsupported major.minor version 52.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)

When i looked this issue #2 , i tried to see my own java version and I have java 1.8 in my computer, so I don't what'is the matter. Can you help me to fix it please?

Higher precision in examples

Divide by Zero Error

Is there any reason why the non-convolutional NN would work perfectly, but when I use a convolutional NN, I get a Divide by Zero exception? It appears to be happening at nn.train(xTrain,yTrain).