teelinsan / kerasdropconnect

An implementation of DropConnect Layer in Keras

Hi there 👋, welcome to my Gihub profile!

I am a PhD student in Computer Science at GLADIA, Sapienza University of Rome (one of the top European universities in AI according to CS ranking). My research activity lies at the intersection of natural language processing, representation learning and machine intelligence.

---

kerasdropconnect's Issues

Need to scale the weights and biases

Hi,
Thanks for the implementation. However, I detected a bug in it.
It uses tf.nn.dropout which scales up the input by 1 / keep_prob (see its documentation). This is not required for dropconnect though. So you have to scale the weights and biases back. Here is the required change (given that the self.prob in you implementation is actually the probability of dropping, not keeping):

self.layer.kernel = K.in_train_phase(K.dropout(self.layer.kernel, self.prob), self.layer.kernel) * (1-self.prob)

self.layer.bias = K.in_train_phase(K.dropout(self.layer.bias, self.prob), self.layer.bias) * (1-self.prob)

Tensorflow and Keras Version

Hi,
I try to use dropcoonect, and already install on my machine,
but when i build and fit model, this error appear

a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
  @tf.function
  def has_init_scope():
    my_constant = tf.constant(1.)
    with tf.init_scope():
      added = my_constant * 2
The graph tensor has name: Model/drop_connect/mul:0

I think there is a mismatch in tensorflow and keras version.
Tensorflow = 2.0.0
Keras = 2.4.0

This is how i use dropconnect

...
X = Dense(64, activation='relu')(X)  
X = DropConnect(layer=Dense(units=32),prob=0.2,)(X)
...

Thanks!