My resume
levithomason / anny Goto Github PK
View Code? Open in Web Editor NEWAnny is an artificial neural network, yo!
Home Page: http://levithomason.github.io/anny/
Anny is an artificial neural network, yo!
Home Page: http://levithomason.github.io/anny/
Currently, we fail under one condition, max epochs have been reached without the error falling below the error threshold. We should also fail if training is resulting in flat or increased error after N epochs.
Momentum will improve learning. Save the previous weight value before updating it as it will be used in the momentum calcuation on the next learning cycle.
There are a few other tricks on this page that may help Anny's performance.
http://andrew.gibiansky.com/blog/machine-learning/convolutional-neural-networks/
Separate layer types for convolution and pooling (max, etc). This is in addition to the regular single dimension layers
Currently, the repo is used to host on gh-pages. This is causing pains by having to ignore dist files, then force add them on the CI process. The master branch then can't be branched for features as it includes ignored files in git cache. We have to maintain an integration branch.
If we setup s3 hosting for instance, we can then simply push deploys there instead and ignore them in the repo.
The network train method is getting large. It also breaks the pattern of the Neuron train and Layer train methods. Neuron.train() update the weights, which makes sense. Layer.train() invokes the Neuron.train() methods which makes sense. Network.train() should invoke Layer.train() methods.
Because the overly complex Network.train
method is there, the method that invokes the Layer.train() methods is awkwardly named correct()
.
The Network.train() method should be pulled into a class, Trainer
. It should return a function that takes in a Network and trains it. It would also house the training options, default callbacks, and any other settings or config related to training (like batch and online training). Then, the Network.correct() method can be renamed more appropriately Network.train(). In the future, it may also support training different types of networks (convolutional, etc).
The Trainer
API may end up looking something like this. It would allow us to have various training strategies.
const shortTrain = new Trainer({maxEpochs: 100})
const accurateTrain = new Trainer({errorThreshold: 0.000001})
// stops training if error is not going down
let lastError = Infinity
const improvingTrain = new Trainer({
onProgress: (error, epoch) => {
if (error > lastError) return false
lastError = error
}
})
// assume we have a net already made
shortTrain(someNetwork)
accurateTrain(someNetwork)
improvingTrain(someNetwork)
These are off the cuff toy examples to demonstrate the pattern
Something like shortTrain can be used to quickly test if a Network can train in a defined amount of time. Accurate train could be used to see if a network could reach a certain level of accuracy. Improving could be used to test how long a network could improve before having regression. All of these trainers could take in a single network config and generate performance stats for a given network in a clean and reusable way.
The trainers could can even be shared as part of challenges. See if your network can "beat the xyz trainer".
Neurons can take an arbitrary input, however, they should pass this value along without modification. Currently, after taking an input the neuron will apply it's activation function to the value before passing it on.
The input value, if present and if this is an input neuron, should be applied directly to the output.
Ref RProp benchmarks found in this paper.
https://github.com/skywinder/github-changelog-generator
This should be added to the deploy step.
Currently, tests are embarrassingly run with no mocks. All unit tests are actually system tests. This means most of the coverage info is exaggerated.
should throw()
tests to test error messages (false positives otherwise)Generate API docs from the existing doc strings.
Simply _.shuffle
the training data samples before training. Especially when using sorted training data, this will result in much better training results.
Currently there is an activation namespace. Its members are activation function definitions. These are objects with func
, prime
, rangeMin
, and rangeMax
properties.
These objects should be created by a class or factory so they can be validated, extended, and created consistently. For better cohesion, this should be a static Neuron class or factory. Same for the namespace, it should likely be a Neuron namespace.
We should likely end up with something along the lines of:
// current namespace with its activations
Neuron.ACTIVATION.tanh
// create new activation
Neuron.ACTIVATION.create({
name: 'newActivation',
func: x => x,
prime: x => x,
rangeMin: 0,
rangeMax: 1,
})
// => Neuron.ACTIVATION.newActivation
EDIT
Also, finish the docs in the Neuron class while at this. There are missing and incomplete doc strings here.
The learning rate should adapt for optimal training. Higher values in the output layer and decreases toward the input layer.
Implement algorithm 4.7 Choosing learning rates (pg 13)
http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf
This is a large library for which we are using only one function, sech
, which is used in only one place ACTIVATION.optimalTanh
. This is quite the overkill. We can easily add a sech
method to the utils, or start our own minimal math util. It might even make sense to simply remove to optimalTanh
entirely for now.
EDIT
Turns out the hyperbolic secant function is simply 1 / Math.cosh(x)
. http://www.mathworks.com/help/matlab/ref/sech.html
Use parallel.js to paralellize training across CPU cores. This shoud also free up rendering during training.
Currently, when a Neuron connect()
s to another, the connection weight can be set. If there is no connection weight specified, random weight initialization is performed based on the number of inputs to the Neuron.
Since the algorithm depends on the number of connections, future connections will cause previously initialized weights to be incorrect since they were based off a fewer number of connections.
The weights must be initialized at a later point in time, prior to training. Ideas for solving this:
Re-init on Neuron connect
When Neuron A connects to Neuron B, B's incoming weights should be re-initialized. A's outgoing do not need re-initialized since the initialization depends only on the number of incoming connections.
Pros
Cons
Init after Layer connect
This would solve many of the cons of the above option. After looping through all the Neurons and making the connections, a final pass through the weights could be made for initialization.
Pros
Cons
initializeWeights()
method
This is the best option so far. The Network could have a method to initialize or randomize its weights. It could make a single pass through all weights and set them based on incoming connection counts (or any other heuristic).
Pros
Cons
new Network
. Then, it would not have to be the default training method and you'd still get the benefits. Sold.Before training, validate the training data.
As of #83, activation ranges are now specified for all activation functions. Using this we can now scale and translate the training data to the optimal range for the activation functions being used in the network.
Creating a layer currently takes a bool to indicate whether or not to include a bias. This should be more declarative, based on network creation, no bias in the output layer.
Currently, Anny utilizes "online training" meaning weights are updated for every sample in the training set. "Batch training" is much more effective (weights are updated after each epoch).
Activation, Error, and Utils should be tested for function names. Ensure that object property names match function name.
Logging on success/fail was never intended to be permanent. It should be updated now. Consider using something like promise syntax for success/fail.
Just as with Neuron activations in #88, there are Network error functions in an ERROR namespace. We should do the same thing and pull these into a class or factory. For better cohesion, it should live on as a static on the Network class. You should be able to create new error functions with validation. When created, they should be added to the Network.ERROR
namespace.
Currently the Neuron.train() method uses the Neuron's error to calculate the delta (used to calculate the gradient and update the weights). This works OK for toy networks with a single output Nueuron. However, the back propagated delta may in fact need to be the derivative of the Neuron's input with respect to the total network error for that training sample, not with respect to the Neuron's particular error. This will represent the weight change needed to affect the total network error, not just that Neuron's error.
Bottom of this page explains it well (now that I know how to work with derivatives, thanks Khan academy!).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.