tushant-yadav / banknotes-authentication-ann

Banknotes-Authentication-ANN

Technical Report Tushant Yadav

Code can be run without any installation using Google Colab. Link at last. Github repository for code. Link at last. The flow of code: Random Initialization of the parameters for a 3 layer neural network. The dimension of neural network layers is [4,10,10,1]. forward propagation part LINEAR part of a layer's forward propagation step. Then we give the ACTIVATION function (relu/Leaky relu/sigmoid/tanh). Combine the previous two steps into a new [LINEAR->ACTIVATION] forward function. Stack the [LINEAR->RELU] forward function L-1 time (for layers 1 through L-1) and add a [LINEAR->SIGMOID] at the end (for the final layer L). This gives us a new L_model_forward function. Compute the loss. Implement the backward propagation module. LINEAR part of a layer's backward propagation step. We get the gradient of the ACTIVATE function (relu_backward, etc.) Combine the previous two steps into a new [LINEAR->ACTIVATION] backward function. Stack [LINEAR->RELU] backward L-1 times and add [LINEAR->SIGMOID] backward in a new L_model_backward function Finally, update the parameters. The weight matrix and biases are updated at every epoch. The model's structure is [LINEAR -> RELU] x (L-1) -> LINEAR -> SIGMOID.

The repository contains 3 files: Helper.py This file consists of all the helper functions required for running of the model. For example activation functions, forward propagation function, etc.
Model.py This file contains the model and its structure once the model is called it uses helper functions to learn and predict. main.py This file contains all the data preprocessing and runs the model for various activation functions.

Results:
Sigmoid

After iteration 0 Training Cost : 0.750625
Testing Cost: 0.746729

After iteration 2900 Training Cost : 0.349225
Testing Cost: 0.368745

training accuracy Accuracy: 0.9260065288356907 test accuracy Accuracy: 0.9183222958057398

F1 score: 0.895184

{'W1': array([[ 1.16724141, 0.10782973, 0.12350197, -0.49559767], [ 0.3916326 , -1.16590349, 0.86205642, -0.385153 ], [ 0.07349732, -0.20950686, 0.71408959, -1.03312616], [-0.12072925, -0.18771349, 0.57854314, -0.53813825], [-0.60257962, -0.27492921, -0.27977425, 0.08625917], [ 0.03056799, 0.68870458, 0.52342941, 0.4541133 ], [ 0.65597319, -0.08532454, -0.0929096 , -0.43957484], [ 0.02745392, 0.30539993, -0.27349231, -0.16782523], [-0.91297654, -0.44221582, -0.47643395, -0.24340374], [-0.475816 , 0.18358287, 0.85897663, 0.3725384 ]]), 'b1': array([[-0.04854884], [ 0.00198593], [-0.00300506], [-0.00466283], [ 0.0833488 ], [-0.11784469], [-0.02351997], [-0.00951234], [ 0.22122949], [ 0.01149 ]]), 'W2': array([[ 0.07615341, -0.28603263, -0.23112969, 0.5026014 , -0.06775241, -0.14568795, 0.10995791, 0.69267904, -0.11859129, 0.16415379], [ 0.30945943, -0.11498507, -0.3622838 , -0.16452429, -0.21483931, 0.26624243, 0.34491995, 0.31707952, -0.18215599, 0.23055035], [-0.62410859, 0.35908464, 0.09204471, -0.07379887, 0.33870257, -0.22503948, 0.17437044, 0.37863709, 1.05690051, -0.4285329 ], [-0.78444128, -0.15999824, 0.05142022, 0.35907167, 0.34260112, -0.7555296 , -0.21152239, 0.2494724 , 0.52411614, 0.31114178], [-0.19846642, -0.06658089, 0.04741957, 0.15004228, 0.13744788, -0.02328778, -0.26493314, 0.09245046, 0.18007365, 0.37389998], [ 0.63504139, 0.05492155, -0.1181718 , -0.26012398, -0.0437195 , 0.13474554, -0.006922 , 0.04609909, -0.51630793, 0.17069331], [-0.01393385, 0.38043729, 0.12578944, 0.15545283, -0.43390158, 0.11106835, 0.28473435, -0.27957212, -0.24012084, -0.01720075], [-0.7300867 , 0.08548125, 0.28656104, -0.18187094, 0.34787618, -0.48349266, -0.1138344 , -0.47792903, 0.79989938, 0.21075872], [ 0.18464787, -0.28811905, 0.33051587, 0.50384814, -0.80044754, 0.43277942, 0.56631076, 0.08087977, -0.72777999, 0.16314701], [-0.32421101, -0.18316777, -0.3758838 , 0.2528269 , 0.44535134, -0.29088921, 0.06789912, -0.38016279, 0.60959031, 0.08528687]]), 'b2': array([[ 0.0155086 ], [ 0.00072759], [-0.11464099], [ 0.01959466], [-0.02136019], [ 0.00585887], [ 0.00559168], [ 0.05653988], [-0.09584261], [ 0.01345663]]), 'W3': array([[-0.25414815, -0.59222352, 0.96160241, 1.05688304, 0.3092424 , -0.76097345, -0.38855105, 1.0770949 , -0.93854914, 0.77049402]]), 'b3': array([[-0.40045649]])}

Leaky ReLu

After iteration 0 Training Cost : 0.774012
Testing Cost: 0.792092

After iteration 2900 Training Cost : 0.012812
Testing Cost: 0.024411

training accuracy Accuracy: 0.976060935799782 test accuracy Accuracy: 0.9735099337748347

F1 score: 0.969697 {'W1': array([[ 2.23330251e+00, 4.82714756e+00, -7.64366683e-01, -2.06140057e+00], [ 3.41058531e-01, -3.90765990e+00, 3.41404138e+00, -1.83036450e-03], [-5.97215028e-02, -2.57005119e+00, 3.01511461e+00, -7.43594154e-01], [-3.10211024e-01, -1.60951756e+00, 1.40819887e+00, -2.54533389e-01], [-6.49872034e-01, -1.76051472e+00, 4.95649174e-01, 7.34954968e-01], [ 1.32825821e+00, 7.25193208e+00, -4.22024976e-01, -1.58468834e+00], [ 7.05391160e-01, -1.42685910e+00, 4.21948038e-01, -7.86629729e-02], [-2.25123756e-01, 4.38942067e+00, -2.26037044e+00, -1.69250990e+00], [-1.02108749e+00, -1.73513171e+00, 1.95251295e-01, 4.16317099e-01], [-9.65243999e-01, -1.18278825e+00, 1.39996451e+00, 7.65097654e-01]]), 'b1': array([[ 0.68327026], [ 0.25555273], [ 0.21063424], [-0.05527848], [-0.06217258], [ 0.74987962], [-0.12549247], [ 0.86506236], [ 0.03070843], [-0.06801504]]), 'W2': array([[-0.19814623, -2.36682846, -1.6893233 , -0.32468804, -0.56603352, -0.31742998, -0.3905863 , 1.25476443, -0.42251281, -0.56728622], [ 0.09202435, -2.1632438 , -1.87985569, -1.03438619, -0.66961765, 0.21902813, -0.15416646, 0.07548172, -0.42045728, -0.61553922], [-2.21618763, 0.26379915, 0.11834878, -0.05485468, 0.2605576 , -2.65796182, 0.25517212, -0.88766802, 0.8378016 , -0.27965045], [-2.57808634, -0.96244917, -0.63077796, -0.01180186, 0.02762961, -3.51259337, -0.38752171, -1.76862839, 0.05919956, 0.14476136], [-2.05532695, -0.29794772, -0.05739652, 0.17909064, 0.20777601, -2.61554243, -0.38231901, -1.15377571, 0.24678228, 0.63589428], [ 0.42182945, -1.87726274, -1.54771426, -1.06827921, -0.4543826 , 0.10991282, -0.48709326, -0.27599625, -0.70138484, -0.64334888], [-1.99856856, 0.49456875, 0.15167342, 0.17355395, -0.41866696, -2.76886598, 0.33068943, -2.44006374, -0.18367062, -0.12156324], [-2.57469072, -0.07261241, 0.08447507, -0.30707116, 0.21259494, -3.32771296, -0.14665319, -2.64198952, 0.47879725, 0.22283183], [ 0.21893802, -0.11408138, 0.41013786, 0.55774185, -0.68623513, 0.54832506, 0.6393158 , -0.45826958, -0.5155148 , 0.04276349], [-2.16002929, -2.24227642, -1.89844067, -0.70980183, -0.32079293, -3.11946746, -0.28469664, -2.48322537, -0.21832463, -0.82370296]]), 'b2': array([[ 0.09560801], [-0.19916325], [ 0.04977254], [-0.10678382], [ 0.12769492], [-0.19432454], [-0.16612573], [-0.16336344], [-0.17541138], [-0.28553272]]), 'W3': array([[ 1.04569805, -0.15083473, 0.70327283, 0.55970861, 0.89714673, -0.4036513 , -0.51122213, 0.46483758, -0.77760467, 0.21925374]]), 'b3': array([[0.34217945]])}

ReLu

After iteration 0 Training Cost : 0.780015
Testing Cost: 0.797278

After iteration 2900 Training Cost : 0.007311
Testing Cost: 0.011961

training accuracy Accuracy: 0.9999999999999998 test accuracy Accuracy: 0.9999999999999998

F1 score: 1.000000 {'W1': array([[ 0.93449001, 0.06935978, 0.28491906, -0.69680935], [ 0.46511762, -1.0983974 , 0.87991582, -0.39974728], [ 0.23561051, -0.09404822, 0.75602171, -1.04028844], [-0.18178871, -0.17268894, 0.65896379, -0.56534916], [-0.30979316, -0.50029176, -0.0043506 , 0.38735392], [ 0.22533377, 1.06039273, 0.64823282, 0.27111253], [ 0.63564673, -0.17312001, 0.20193516, -0.48469902], [-0.40623374, 0.25562777, -0.76237373, -0.29516999], [-0.77518291, -0.31083619, -0.24952341, -0.06952035], [-0.74427899, 0.12695384, 0.84714965, 0.37099277]]), 'b1': array([[-0.01645378], [-0.01854571], [ 0.016558 ], [ 0.07678154], [ 0.0996241 ], [-0.16223049], [ 0.01801708], [ 0.32505409], [ 0.27360111], [ 0.10263598]]), 'W2': array([[-0.06145903, -0.24430075, -0.23267234, 0.58547026, 0.06098987, -0.33041455, 0.07505478, 0.7977329 , 0.10833693, 0.30572624], [ 0.19302312, -0.09106625, -0.36266282, -0.15942478, -0.08400067, 0.37698598, 0.28351315, 0.21698497, 0.05038635, 0.19956984], [-0.37083204, 0.30727067, 0.10327626, -0.05340179, 0.20551076, -0.18967726, 0.30442924, 0.57610828, 0.77751517, -0.30769029], [-0.43333495, -0.07944202, 0.04224601, 0.37759188, 0.25066632, -0.60706344, -0.09252331, 0.71330118, 0.26046672, 0.51429441], [-0.1444562 , -0.05497069, -0.01468957, 0.21516075, 0.18418683, -0.21294295, -0.22884834, 0.34376713, 0.2005074 , 0.5793896 ], [ 0.52868736, 0.13099349, -0.01261763, -0.19620705, 0.09304122, 0.29255139, -0.05008451, -0.01740382, -0.25140388, 0.13189702], [ 0.08751448, 0.50157272, 0.23923375, 0.19647524, -0.39988941, 0.26720761, 0.35252195, -0.26339528, -0.15031335, -0.03140074], [-0.48473956, 0.12195169, 0.21697199, -0.26286724, 0.20108757, -0.40692155, -0.04932102, -0.50345725, 0.44314009, 0.22689415], [ 0.24535056, -0.0503129 , 0.51925658, 0.59973513, -0.61475813, 0.77060374, 0.62078138, -0.01870784, -0.45867964, 0.13594312], [-0.05729268, -0.13984637, -0.37918812, 0.19602697, 0.29088326, -0.18310905, 0.16902638, -0.35704595, 0.28987317, 0.06853523]]), 'b2': array([[ 0.15903723], [-0.06619995], [ 0.12016296], [ 0.36718604], [ 0.23988325], [-0.04400087], [ 0.03759301], [ 0.05996689], [-0.06145823], [ 0.04952392]]), 'W3': array([[ 0.63229243, -0.28359062, 0.48195515, 0.96798239, 0.74754177, -0.44405631, -0.55573471, 0.43069036, -0.82835618, 0.26710055]]), 'b3': array([[0.20505645]])}

Tanh

After iteration 0 Training Cost : 0.581206
Testing Cost: 0.601567

After iteration 2900 Training Cost : 0.589392
Testing Cost: 0.609148

training accuracy Accuracy: 0.44069640914036995 test accuracy Accuracy: 0.43046357615894043

F1 score: 0.601852 {'W1': array([[ 0.61957879, -1.28481441, -0.9785516 , 0.08259855], [ 0.26453886, -2.01622957, 0.24135984, 0.16785965], [-0.06474247, -1.26808197, -0.10337144, -0.30690497], [-0.22602828, -0.52477099, 0.32426312, -0.33918784], [-0.21262145, -1.09319302, -0.4556971 , 0.70659109], [-0.78105907, -0.62652759, -0.4223529 , 1.01238509], [ 0.41133382, -0.54473097, -0.20921469, -0.3390683 ], [-0.22236992, -0.18836259, -0.67661695, 0.08893628], [-0.3699886 , -0.56274562, -0.43737605, 0.08314269], [-0.62024181, -0.19453814, 0.60229505, 0.56789665]]), 'b1': array([[-0.50925927], [-0.45069113], [-0.59483214], [-0.17322518], [-0.34094214], [-0.62487259], [-0.10574815], [-0.23614306], [-0.07324517], [-0.16204694]]), 'W2': array([[15.31042842, 9.15392086, 11.81361245, 7.58115774, 12.02130003, 15.45840703, 8.5342804 , 18.13692426, 13.96087217, 6.82235437], [15.45419198, 9.31344629, 11.67741891, 6.92930182, 11.92895474, 15.83310443, 8.73117403, 17.76136247, 14.00417867, 6.90136146], [14.55349405, 9.35485638, 11.67152383, 6.65928688, 11.66357669, 15.05636313, 8.44274738, 17.66585987, 14.17678394, 5.91300078], [14.71883405, 9.11407915, 11.91855319, 7.22600475, 11.93759921, 14.82794213, 8.24601718, 17.63879054, 13.84855384, 6.77744165], [15.03009406, 9.1486083 , 11.85608637, 7.03913667, 11.83638843, 15.42608115, 8.07992873, 17.45895188, 13.75803148, 6.85731827], [15.73160076, 9.47786563, 11.913759 , 6.83448131, 12.12325324, 15.66535429, 8.35261752, 17.47745075, 13.71290785, 6.83913051], [15.2177612 , 9.81202867, 12.16626422, 7.22753432, 11.64879192, 15.7011938 , 8.7000296 , 17.16533449, 13.82976875, 6.63182811], [14.79317067, 9.41598995, 12.18367573, 6.7026 , 11.99310094, 15.1027113 , 8.36528516, 16.89153885, 14.16957009, 6.68969354], [15.36500696, 9.19080461, 12.45425577, 7.66888707, 11.41902474, 16.05272636, 8.98971827, 17.58050504, 13.54478223, 6.90098249], [15.18963957, 9.1413062 , 11.54550177, 7.15875948, 12.14956814, 15.34108601, 8.55534863, 17.05023185, 14.08305263, 6.58453114]]), 'b2': array([[-29.9955253 ], [-29.97684001], [-29.05338431], [-29.67320341], [-29.55579661], [-29.96565975], [-29.97650894], [-29.75959654], [-29.99810989], [-29.78986977]]), 'W3': array([[ 0.01828437, 0.09911072, 0.47220322, 0.24718742, 0.31441874, -0.08219113, -0.07267266, 0.20870558, 0.00097718, 0.18948714]]), 'b3': array([[0.78390117]])}

Colab notebook link:- https://colab.research.google.com/drive/1XGXsLRaEvqBUVuKHb4a3cTd-DTXLkl04#scrollTo=sgDHQvrYV4V2 Github repository link:- https://github.com/tushant-yadav/Banknotes-Authentication-ANN