katsura-jp / efficientnet-pytorch Goto Github PK

May be I might be wrong, however I found that your MBConv block dosen't have skip connection.

In your model it's

MBConv

compare to

https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py

MBConvBlock class

Or maybe not :(

I use the net to train cifar-100, but the after 300 epoch the train_error reaches to 0 but the test_error is still about 55%（acually it reach 55% after 50 epoch）. The paper said that the effcientnet for cifar-100 accuracy can reach 88.1%. What's the problem?
I'm using efficientnet-b3 and the optimizer is torch.optim.SGD(model.parameters(),lr=0.1.momentum=0.9,weight_decay=1e-5)

I simply download your code and run cifar100.py on K80 with
torch == 1.1.0
torchvision == 0.2.2
But the trian and test error is still high even having been trained by 100 epochs.
Did I do anything wrong?

Following is the output of terminal
Files already downloaded and verified
Files already downloaded and verified
======== epoch 0 (lr: 0.01600) ========
train loss = 4.75042 | train err = 98.99% |
test loss = 4.60554 | test err = 99.01%
======== epoch 1 (lr: 0.01600) ========
train loss = 4.60711 | train err = 99.05% |
test loss = 4.60681 | test err = 99.01%
======== epoch 2 (lr: 0.01552) ========
train loss = 4.60656 | train err = 98.97% |
test loss = 4.60671 | test err = 99.01%
======== epoch 3 (lr: 0.01552) ========
train loss = 4.60680 | train err = 99.07% |
test loss = 4.60664 | test err = 99.01%
======== epoch 4 (lr: 0.01505) ========
train loss = 4.60665 | train err = 98.95% |
test loss = 4.60659 | test err = 99.01%
======== epoch 5 (lr: 0.01505) ========
train loss = 4.60670 | train err = 98.98% |
test loss = 4.60672 | test err = 99.01%
======== epoch 6 (lr: 0.01505) ========
train loss = 4.60649 | train err = 99.08% |
test loss = 4.60659 | test err = 99.01%
======== epoch 7 (lr: 0.01460) ========
train loss = 4.60658 | train err = 99.04% |
test loss = 4.60645 | test err = 99.01%
======== epoch 8 (lr: 0.01460) ========
train loss = 4.60652 | train err = 99.03% |
test loss = 4.60651 | test err = 98.94%
======== epoch 9 (lr: 0.01416) ========
train loss = 4.60657 | train err = 99.11% |
test loss = 4.60653 | test err = 99.01%
======== epoch 10 (lr: 0.01416) ========
train loss = 4.60649 | train err = 98.97% |
test loss = 4.60653 | test err = 99.01%
======== epoch 11 (lr: 0.01374) ========
train loss = 4.60658 | train err = 98.96% |
test loss = 4.60603 | test err = 99.01%
======== epoch 12 (lr: 0.01374) ========
train loss = 4.60644 | train err = 99.00% |
test loss = 4.60641 | test err = 99.01%
======== epoch 13 (lr: 0.01374) ========
train loss = 4.60625 | train err = 99.00% |
test loss = 4.60647 | test err = 99.01%
======== epoch 14 (lr: 0.01333) ========
train loss = 4.60643 | train err = 99.04% |
test loss = 4.60629 | test err = 99.01%
======== epoch 15 (lr: 0.01333) ========
train loss = 4.60619 | train err = 99.02% |
test loss = 4.60632 | test err = 99.01%
======== epoch 16 (lr: 0.01293) ========
train loss = 4.60616 | train err = 99.04% |
test loss = 4.60624 | test err = 99.01%
======== epoch 17 (lr: 0.01293) ========
train loss = 4.60627 | train err = 99.00% |
test loss = 4.60632 | test err = 99.01%
======== epoch 18 (lr: 0.01293) ========
train loss = 4.60628 | train err = 98.98% |
test loss = 4.60630 | test err = 99.01%
======== epoch 19 (lr: 0.01254) ========
train loss = 4.60628 | train err = 99.00% |
test loss = 4.60613 | test err = 99.01%
======== epoch 20 (lr: 0.01254) ========
train loss = 4.60622 | train err = 98.94% |
test loss = 4.60623 | test err = 99.01%
======== epoch 21 (lr: 0.01216) ========
train loss = 4.60622 | train err = 99.05% |
test loss = 4.60608 | test err = 99.01%
======== epoch 22 (lr: 0.01216) ========
train loss = 4.60613 | train err = 98.93% |
test loss = 4.60622 | test err = 99.01%
======== epoch 23 (lr: 0.01180) ========
train loss = 4.60618 | train err = 98.95% |
test loss = 4.60577 | test err = 99.01%
======== epoch 24 (lr: 0.01180) ========
train loss = 4.60615 | train err = 99.02% |
test loss = 4.60619 | test err = 99.01%
======== epoch 25 (lr: 0.01180) ========
train loss = 4.60616 | train err = 99.00% |
test loss = 4.60606 | test err = 99.01%
======== epoch 26 (lr: 0.01144) ========
train loss = 4.60617 | train err = 99.04% |
test loss = 4.60606 | test err = 99.01%
======== epoch 27 (lr: 0.01144) ========
train loss = 4.60593 | train err = 98.94% |
test loss = 4.60610 | test err = 99.01%
======== epoch 28 (lr: 0.01110) ========
train loss = 4.60595 | train err = 99.03% |
test loss = 4.60600 | test err = 99.01%
======== epoch 29 (lr: 0.01110) ========
train loss = 4.60597 | train err = 99.02% |
test loss = 4.60595 | test err = 99.01%
======== epoch 30 (lr: 0.01110) ========
train loss = 4.60599 | train err = 98.91% |
test loss = 4.60603 | test err = 99.01%
======== epoch 31 (lr: 0.01077) ========
train loss = 4.60592 | train err = 99.00% |
test loss = 4.60591 | test err = 98.94%
======== epoch 32 (lr: 0.01077) ========
train loss = 4.60596 | train err = 99.00% |
test loss = 4.60596 | test err = 99.01%
======== epoch 33 (lr: 0.01045) ========
train loss = 4.60598 | train err = 98.92% |
test loss = 4.60590 | test err = 99.01%
======== epoch 34 (lr: 0.01045) ========
train loss = 4.60598 | train err = 98.89% |
test loss = 4.60590 | test err = 99.01%
======== epoch 35 (lr: 0.01013) ========
train loss = 4.60594 | train err = 99.03% |
test loss = 4.60560 | test err = 99.01%
======== epoch 36 (lr: 0.01013) ========
train loss = 4.60595 | train err = 99.08% |
test loss = 4.60580 | test err = 98.94%
======== epoch 37 (lr: 0.01013) ========
train loss = 4.60583 | train err = 98.92% |
test loss = 4.60591 | test err = 99.01%
======== epoch 38 (lr: 0.00983) ========
train loss = 4.60586 | train err = 99.00% |
test loss = 4.60580 | test err = 98.94%
======== epoch 39 (lr: 0.00983) ========
train loss = 4.60583 | train err = 99.02% |
test loss = 4.60581 | test err = 99.01%
======== epoch 40 (lr: 0.00953) ========
train loss = 4.60594 | train err = 98.94% |
test loss = 4.60580 | test err = 99.01%
======== epoch 41 (lr: 0.00953) ========
train loss = 4.60585 | train err = 99.07% |
test loss = 4.60586 | test err = 99.01%
======== epoch 42 (lr: 0.00953) ========
train loss = 4.60593 | train err = 99.06% |
test loss = 4.60571 | test err = 99.01%
======== epoch 43 (lr: 0.00925) ========
train loss = 4.60587 | train err = 98.96% |
test loss = 4.60572 | test err = 99.01%
======== epoch 44 (lr: 0.00925) ========
train loss = 4.60573 | train err = 98.91% |
test loss = 4.60576 | test err = 98.94%
======== epoch 45 (lr: 0.00897) ========
train loss = 4.60579 | train err = 99.09% |
test loss = 4.60572 | test err = 99.01%
======== epoch 46 (lr: 0.00897) ========
train loss = 4.60568 | train err = 99.05% |
test loss = 4.60574 | test err = 99.01%
======== epoch 47 (lr: 0.00870) ========
train loss = 4.60585 | train err = 99.04% |
test loss = 4.60555 | test err = 99.01%
======== epoch 48 (lr: 0.00870) ========
train loss = 4.60557 | train err = 98.98% |
test loss = 4.60565 | test err = 98.94%
======== epoch 49 (lr: 0.00870) ========
train loss = 4.60570 | train err = 98.96% |
test loss = 4.60565 | test err = 98.94%
======== epoch 50 (lr: 0.00844) ========
train loss = 4.60566 | train err = 99.02% |
test loss = 4.60564 | test err = 99.01%
======== epoch 51 (lr: 0.00844) ========
train loss = 4.60564 | train err = 99.01% |
test loss = 4.60572 | test err = 99.01%
======== epoch 52 (lr: 0.00819) ========
train loss = 4.60576 | train err = 99.01% |
test loss = 4.60564 | test err = 99.01%
======== epoch 53 (lr: 0.00819) ========
train loss = 4.60566 | train err = 99.07% |
test loss = 4.60567 | test err = 99.01%
======== epoch 54 (lr: 0.00819) ========
train loss = 4.60566 | train err = 98.99% |
test loss = 4.60572 | test err = 99.01%
======== epoch 55 (lr: 0.00794) ========
train loss = 4.60581 | train err = 99.03% |
test loss = 4.60558 | test err = 98.94%
======== epoch 56 (lr: 0.00794) ========
train loss = 4.60555 | train err = 98.98% |
test loss = 4.60566 | test err = 99.01%
======== epoch 57 (lr: 0.00770) ========
train loss = 4.60555 | train err = 99.00% |
test loss = 4.60560 | test err = 99.01%
======== epoch 58 (lr: 0.00770) ========
train loss = 4.60575 | train err = 99.04% |
test loss = 4.60568 | test err = 99.01%
======== epoch 59 (lr: 0.00747) ========
train loss = 4.60564 | train err = 99.01% |
test loss = 4.60546 | test err = 99.01%
======== epoch 60 (lr: 0.00747) ========
train loss = 4.60566 | train err = 99.06% |
test loss = 4.60559 | test err = 99.01%
======== epoch 61 (lr: 0.00747) ========
train loss = 4.60559 | train err = 98.99% |
test loss = 4.60558 | test err = 99.01%
======== epoch 62 (lr: 0.00725) ========
train loss = 4.60551 | train err = 99.06% |
test loss = 4.60557 | test err = 99.01%
======== epoch 63 (lr: 0.00725) ========
train loss = 4.60566 | train err = 98.99% |
test loss = 4.60555 | test err = 99.01%
======== epoch 64 (lr: 0.00703) ========
train loss = 4.60563 | train err = 99.03% |
test loss = 4.60562 | test err = 99.01%
======== epoch 65 (lr: 0.00703) ========
train loss = 4.60559 | train err = 99.04% |
test loss = 4.60551 | test err = 98.94%
======== epoch 66 (lr: 0.00703) ========
train loss = 4.60555 | train err = 99.02% |
test loss = 4.60549 | test err = 99.01%
======== epoch 67 (lr: 0.00682) ========
train loss = 4.60547 | train err = 99.04% |
test loss = 4.60552 | test err = 99.01%
======== epoch 68 (lr: 0.00682) ========
train loss = 4.60565 | train err = 99.06% |
test loss = 4.60559 | test err = 99.01%
======== epoch 69 (lr: 0.00661) ========
train loss = 4.60551 | train err = 98.94% |
test loss = 4.60550 | test err = 98.94%
======== epoch 70 (lr: 0.00661) ========
train loss = 4.60546 | train err = 98.97% |
test loss = 4.60552 | test err = 99.01%
======== epoch 71 (lr: 0.00642) ========
train loss = 4.60546 | train err = 98.97% |
test loss = 4.60539 | test err = 99.01%
======== epoch 72 (lr: 0.00642) ========
train loss = 4.60555 | train err = 99.07% |
test loss = 4.60548 | test err = 99.01%
======== epoch 73 (lr: 0.00642) ========
train loss = 4.60542 | train err = 98.97% |
test loss = 4.60550 | test err = 99.01%
======== epoch 74 (lr: 0.00622) ========
train loss = 4.60550 | train err = 98.92% |
test loss = 4.60546 | test err = 99.01%
======== epoch 75 (lr: 0.00622) ========
train loss = 4.60547 | train err = 98.98% |
test loss = 4.60545 | test err = 99.01%
======== epoch 76 (lr: 0.00604) ========
train loss = 4.60542 | train err = 99.03% |
test loss = 4.60539 | test err = 99.01%
======== epoch 77 (lr: 0.00604) ========
train loss = 4.60546 | train err = 98.98% |
test loss = 4.60546 | test err = 99.01%
======== epoch 78 (lr: 0.00604) ========
train loss = 4.60552 | train err = 99.06% |
test loss = 4.60551 | test err = 99.01%
======== epoch 79 (lr: 0.00586) ========
train loss = 4.60542 | train err = 99.00% |
test loss = 4.60548 | test err = 99.01%
======== epoch 80 (lr: 0.00586) ========
train loss = 4.60553 | train err = 99.01% |
test loss = 4.60543 | test err = 99.01%
======== epoch 81 (lr: 0.00568) ========
train loss = 4.60548 | train err = 99.00% |
test loss = 4.60545 | test err = 99.01%
======== epoch 82 (lr: 0.00568) ========
train loss = 4.60537 | train err = 98.94% |
test loss = 4.60541 | test err = 99.01%
======== epoch 83 (lr: 0.00551) ========
train loss = 4.60549 | train err = 98.97% |
test loss = 4.60538 | test err = 99.01%
======== epoch 84 (lr: 0.00551) ========
train loss = 4.60545 | train err = 99.09% |
test loss = 4.60545 | test err = 99.01%
======== epoch 85 (lr: 0.00551) ========
train loss = 4.60553 | train err = 99.07% |
test loss = 4.60543 | test err = 99.01%
======== epoch 86 (lr: 0.00534) ========
train loss = 4.60541 | train err = 98.98% |
test loss = 4.60543 | test err = 99.01%
======== epoch 87 (lr: 0.00534) ========
train loss = 4.60542 | train err = 99.03% |
test loss = 4.60543 | test err = 99.01%
======== epoch 88 (lr: 0.00518) ========
train loss = 4.60545 | train err = 98.98% |
test loss = 4.60537 | test err = 99.01%
======== epoch 89 (lr: 0.00518) ========
train loss = 4.60553 | train err = 98.91% |
test loss = 4.60538 | test err = 99.01%
======== epoch 90 (lr: 0.00518) ========
train loss = 4.60535 | train err = 98.96% |
test loss = 4.60542 | test err = 99.01%
======== epoch 91 (lr: 0.00503) ========
train loss = 4.60539 | train err = 98.98% |
test loss = 4.60541 | test err = 99.01%
======== epoch 92 (lr: 0.00503) ========
train loss = 4.60538 | train err = 99.01% |
test loss = 4.60545 | test err = 99.01%
======== epoch 93 (lr: 0.00488) ========
train loss = 4.60539 | train err = 99.02% |
test loss = 4.60539 | test err = 99.01%
======== epoch 94 (lr: 0.00488) ========
train loss = 4.60539 | train err = 99.01% |
test loss = 4.60533 | test err = 99.01%
======== epoch 95 (lr: 0.00473) ========
train loss = 4.60539 | train err = 98.97% |
test loss = 4.60533 | test err = 99.01%
======== epoch 96 (lr: 0.00473) ========
train loss = 4.60532 | train err = 98.95% |
test loss = 4.60539 | test err = 99.01%
======== epoch 97 (lr: 0.00473) ========
train loss = 4.60537 | train err = 99.05% |
test loss = 4.60535 | test err = 98.94%
======== epoch 98 (lr: 0.00459) ========
train loss = 4.60541 | train err = 99.00% |
test loss = 4.60537 | test err = 99.01%
======== epoch 99 (lr: 0.00459) ========
train loss = 4.60544 | train err = 98.99% |
test loss = 4.60538 | test err = 99.01%
======== epoch 100 (lr: 0.00445) ========
train loss = 4.60533 | train err = 98.96% |
test loss = 4.60536 | test err = 99.01%
======== epoch 101 (lr: 0.00445) ========
train loss = 4.60529 | train err = 98.93% |
test loss = 4.60533 | test err = 99.01%
======== epoch 102 (lr: 0.00445) ========
train loss = 4.60531 | train err = 98.98% |
test loss = 4.60536 | test err = 99.01%
======== epoch 103 (lr: 0.00432) ========
train loss = 4.60539 | train err = 99.12% |
test loss = 4.60530 | test err = 98.94%

According to the article Swish should be input * torch.sigmoid(beta * input), where beta - trainable.
Your realization has no beta param.

i cant understand the test_err 99.01% on epoch 1.
Sorry
I have two quenstions.
(1)
i think it is too good for cifar100.
Why?

(2)
(When I tried to do cifar10 by modifying your source codes, the test_err was about 90% on epoch 1.
Why is cifar10_value worse than cifar100?

@katsura-jp I'm wondering why there is upsample operation in before stem ?