Training speed become slower iteration by iteration about faster-rcnn_tf

[philokey]

I find the reason.
In training processing, the speed of the following code will slow down (I still do not know why).

if iter >= cfg.TRAIN.STEPSIZE:
     sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE * cfg.TRAIN.GAMMA))
else:
    sess.run(tf.assign(lr, cfg.TRAIN.LEARNING_RATE))

Hence, set the learning rate out of the loop can avoid this problem.

[flowice]

@philokey Hi, I also find the problem. What do you mean by "set the learning rate out of the loop can avoid this problem"? I think it is weird. Have you found the reason?

[philokey]

@flowice when you set learning rate, it will add a new node in tensorflow's graph, therefore, if you set learning rate in every iteration, it will add many nodes in the graph and become very slow.

[flowice]

@philokey Wonderful ! I will have a try. Thank you.