The training speed of gluon-ssd is slower than that of mxnet-ssd (39 vs 120 images / second) about gluon-cv

[zhreshold]

The synchronization in loss is preferred, without it, mAP is reduced by ~1%.
The sampler is now pushed to multiple CPU worker.

[zhreshold]

I think you are correct, a minor change during the experiment brake the sychronization therefore target generator is not in parallel.

I will try to fix it ASAP, thanks

[wkcn]

@zhreshold Thank you! I will try to replace Block with Hybrid-Block too.

[wkcn]

I found the reason is that there are calculation operation (like:asnumpy or wait_to_all) in some Block.
The ugly solution is to write a mx.sym.CustomOP to encapsulate these operations.

I have replaced bloxk with hybrid block. https://github.com/wkcn/gluon-cv/tree/improve_ssd_speed
However, it seems that there Is a dead lock in gluon.DataLoader. I couldn't test it.

[zhreshold]

A better way to do it is to move target generator to data transform since training image size is fixed and anchors are fixed therefore. One tricky stuff is negative mining and positive sample synchronizing across devices. I will investigate more to finalize a solution to it.

[wkcn]

Great. Thank you!
I have improve the training speed to 60~70 images / second using Tesla M40 x 4.
And there may be no any improvement to hybridize target_generator. (I tried it on the rebuilding MXNet, no cudnn, fixed the box_iou operator bug)

[zhreshold]

See #99

Since synchronization is still required, it won't get linear speed-up, but definitely getting faster.

[wkcn]

@zhreshold I will try it.
Thank you!

In my test, the training speed of the re-organized code(e9cc2bf) is still about 45 samples/sec.
I think the bottleneck is that there are asnumpy and asscalar in the mx.gluon.Block.
It leads to serial calculation rather than serial calculation.

https://github.com/dmlc/gluon-cv/blob/master/gluoncv/utils/nn/sampler.py#L78
https://github.com/dmlc/gluon-cv/blob/master/gluoncv/loss.py#L133

[wkcn]

@zhreshold
Hi.
In loss.py line 133,
Change

        for cp, bp, ct, bt in zip(*[cls_pred, box_pred, cls_target, box_target]):
            pos_samples = (ct > 0)
            num_pos.append(pos_samples.sum().asscalar())
        num_pos_all = sum(num_pos)

to

        for cp, bp, ct, bt in zip(*[cls_pred, box_pred, cls_target, box_target]):
            pos_samples = (ct > 0)
            num_pos.append(pos_samples.sum())
        nd.waitall()
        num_pos_all = sum([p.asscalar() for p in num_pos])

I think the latter is faster.

[zhreshold]

Sounds good, but I think nd.waitall() may not be necessary.
Do you have numbers regarding this? @wkcn

[wkcn]

@zhreshold I do an experiment and found that nd.waitall() is not necessary.
When an asscalar() is called, computational flow graph in all devices will be computed.

[wkcn]

The speed of the latest implementation is 102 samples / sec in M40 x 4.
Great! Thank you!