katsura-jp / efficientnet-pytorch Goto Github PK
View Code? Open in Web Editor NEWPyTorch implementation of "EfficientNet", ICML 2019
PyTorch implementation of "EfficientNet", ICML 2019
May be I might be wrong, however I found that your MBConv block dosen't have skip connection.
In your model it's
MBConv
compare to
https://github.com/lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/model.py
MBConvBlock class
Or maybe not :(
I use the net to train cifar-100, but the after 300 epoch the train_error reaches to 0 but the test_error is still about 55%(acually it reach 55% after 50 epoch). The paper said that the effcientnet for cifar-100 accuracy can reach 88.1%. What's the problem?
I'm using efficientnet-b3 and the optimizer is torch.optim.SGD(model.parameters(),lr=0.1.momentum=0.9,weight_decay=1e-5)
I simply download your code and run cifar100.py on K80 with
torch == 1.1.0
torchvision == 0.2.2
But the trian and test error is still high even having been trained by 100 epochs.
Did I do anything wrong?
Following is the output of terminal
Files already downloaded and verified
Files already downloaded and verified
======== epoch 0 (lr: 0.01600) ========
train loss = 4.75042 | train err = 98.99% |
test loss = 4.60554 | test err = 99.01%
======== epoch 1 (lr: 0.01600) ========
train loss = 4.60711 | train err = 99.05% |
test loss = 4.60681 | test err = 99.01%
======== epoch 2 (lr: 0.01552) ========
train loss = 4.60656 | train err = 98.97% |
test loss = 4.60671 | test err = 99.01%
======== epoch 3 (lr: 0.01552) ========
train loss = 4.60680 | train err = 99.07% |
test loss = 4.60664 | test err = 99.01%
======== epoch 4 (lr: 0.01505) ========
train loss = 4.60665 | train err = 98.95% |
test loss = 4.60659 | test err = 99.01%
======== epoch 5 (lr: 0.01505) ========
train loss = 4.60670 | train err = 98.98% |
test loss = 4.60672 | test err = 99.01%
======== epoch 6 (lr: 0.01505) ========
train loss = 4.60649 | train err = 99.08% |
test loss = 4.60659 | test err = 99.01%
======== epoch 7 (lr: 0.01460) ========
train loss = 4.60658 | train err = 99.04% |
test loss = 4.60645 | test err = 99.01%
======== epoch 8 (lr: 0.01460) ========
train loss = 4.60652 | train err = 99.03% |
test loss = 4.60651 | test err = 98.94%
======== epoch 9 (lr: 0.01416) ========
train loss = 4.60657 | train err = 99.11% |
test loss = 4.60653 | test err = 99.01%
======== epoch 10 (lr: 0.01416) ========
train loss = 4.60649 | train err = 98.97% |
test loss = 4.60653 | test err = 99.01%
======== epoch 11 (lr: 0.01374) ========
train loss = 4.60658 | train err = 98.96% |
test loss = 4.60603 | test err = 99.01%
======== epoch 12 (lr: 0.01374) ========
train loss = 4.60644 | train err = 99.00% |
test loss = 4.60641 | test err = 99.01%
======== epoch 13 (lr: 0.01374) ========
train loss = 4.60625 | train err = 99.00% |
test loss = 4.60647 | test err = 99.01%
======== epoch 14 (lr: 0.01333) ========
train loss = 4.60643 | train err = 99.04% |
test loss = 4.60629 | test err = 99.01%
======== epoch 15 (lr: 0.01333) ========
train loss = 4.60619 | train err = 99.02% |
test loss = 4.60632 | test err = 99.01%
======== epoch 16 (lr: 0.01293) ========
train loss = 4.60616 | train err = 99.04% |
test loss = 4.60624 | test err = 99.01%
======== epoch 17 (lr: 0.01293) ========
train loss = 4.60627 | train err = 99.00% |
test loss = 4.60632 | test err = 99.01%
======== epoch 18 (lr: 0.01293) ========
train loss = 4.60628 | train err = 98.98% |
test loss = 4.60630 | test err = 99.01%
======== epoch 19 (lr: 0.01254) ========
train loss = 4.60628 | train err = 99.00% |
test loss = 4.60613 | test err = 99.01%
======== epoch 20 (lr: 0.01254) ========
train loss = 4.60622 | train err = 98.94% |
test loss = 4.60623 | test err = 99.01%
======== epoch 21 (lr: 0.01216) ========
train loss = 4.60622 | train err = 99.05% |
test loss = 4.60608 | test err = 99.01%
======== epoch 22 (lr: 0.01216) ========
train loss = 4.60613 | train err = 98.93% |
test loss = 4.60622 | test err = 99.01%
======== epoch 23 (lr: 0.01180) ========
train loss = 4.60618 | train err = 98.95% |
test loss = 4.60577 | test err = 99.01%
======== epoch 24 (lr: 0.01180) ========
train loss = 4.60615 | train err = 99.02% |
test loss = 4.60619 | test err = 99.01%
======== epoch 25 (lr: 0.01180) ========
train loss = 4.60616 | train err = 99.00% |
test loss = 4.60606 | test err = 99.01%
======== epoch 26 (lr: 0.01144) ========
train loss = 4.60617 | train err = 99.04% |
test loss = 4.60606 | test err = 99.01%
======== epoch 27 (lr: 0.01144) ========
train loss = 4.60593 | train err = 98.94% |
test loss = 4.60610 | test err = 99.01%
======== epoch 28 (lr: 0.01110) ========
train loss = 4.60595 | train err = 99.03% |
test loss = 4.60600 | test err = 99.01%
======== epoch 29 (lr: 0.01110) ========
train loss = 4.60597 | train err = 99.02% |
test loss = 4.60595 | test err = 99.01%
======== epoch 30 (lr: 0.01110) ========
train loss = 4.60599 | train err = 98.91% |
test loss = 4.60603 | test err = 99.01%
======== epoch 31 (lr: 0.01077) ========
train loss = 4.60592 | train err = 99.00% |
test loss = 4.60591 | test err = 98.94%
======== epoch 32 (lr: 0.01077) ========
train loss = 4.60596 | train err = 99.00% |
test loss = 4.60596 | test err = 99.01%
======== epoch 33 (lr: 0.01045) ========
train loss = 4.60598 | train err = 98.92% |
test loss = 4.60590 | test err = 99.01%
======== epoch 34 (lr: 0.01045) ========
train loss = 4.60598 | train err = 98.89% |
test loss = 4.60590 | test err = 99.01%
======== epoch 35 (lr: 0.01013) ========
train loss = 4.60594 | train err = 99.03% |
test loss = 4.60560 | test err = 99.01%
======== epoch 36 (lr: 0.01013) ========
train loss = 4.60595 | train err = 99.08% |
test loss = 4.60580 | test err = 98.94%
======== epoch 37 (lr: 0.01013) ========
train loss = 4.60583 | train err = 98.92% |
test loss = 4.60591 | test err = 99.01%
======== epoch 38 (lr: 0.00983) ========
train loss = 4.60586 | train err = 99.00% |
test loss = 4.60580 | test err = 98.94%
======== epoch 39 (lr: 0.00983) ========
train loss = 4.60583 | train err = 99.02% |
test loss = 4.60581 | test err = 99.01%
======== epoch 40 (lr: 0.00953) ========
train loss = 4.60594 | train err = 98.94% |
test loss = 4.60580 | test err = 99.01%
======== epoch 41 (lr: 0.00953) ========
train loss = 4.60585 | train err = 99.07% |
test loss = 4.60586 | test err = 99.01%
======== epoch 42 (lr: 0.00953) ========
train loss = 4.60593 | train err = 99.06% |
test loss = 4.60571 | test err = 99.01%
======== epoch 43 (lr: 0.00925) ========
train loss = 4.60587 | train err = 98.96% |
test loss = 4.60572 | test err = 99.01%
======== epoch 44 (lr: 0.00925) ========
train loss = 4.60573 | train err = 98.91% |
test loss = 4.60576 | test err = 98.94%
======== epoch 45 (lr: 0.00897) ========
train loss = 4.60579 | train err = 99.09% |
test loss = 4.60572 | test err = 99.01%
======== epoch 46 (lr: 0.00897) ========
train loss = 4.60568 | train err = 99.05% |
test loss = 4.60574 | test err = 99.01%
======== epoch 47 (lr: 0.00870) ========
train loss = 4.60585 | train err = 99.04% |
test loss = 4.60555 | test err = 99.01%
======== epoch 48 (lr: 0.00870) ========
train loss = 4.60557 | train err = 98.98% |
test loss = 4.60565 | test err = 98.94%
======== epoch 49 (lr: 0.00870) ========
train loss = 4.60570 | train err = 98.96% |
test loss = 4.60565 | test err = 98.94%
======== epoch 50 (lr: 0.00844) ========
train loss = 4.60566 | train err = 99.02% |
test loss = 4.60564 | test err = 99.01%
======== epoch 51 (lr: 0.00844) ========
train loss = 4.60564 | train err = 99.01% |
test loss = 4.60572 | test err = 99.01%
======== epoch 52 (lr: 0.00819) ========
train loss = 4.60576 | train err = 99.01% |
test loss = 4.60564 | test err = 99.01%
======== epoch 53 (lr: 0.00819) ========
train loss = 4.60566 | train err = 99.07% |
test loss = 4.60567 | test err = 99.01%
======== epoch 54 (lr: 0.00819) ========
train loss = 4.60566 | train err = 98.99% |
test loss = 4.60572 | test err = 99.01%
======== epoch 55 (lr: 0.00794) ========
train loss = 4.60581 | train err = 99.03% |
test loss = 4.60558 | test err = 98.94%
======== epoch 56 (lr: 0.00794) ========
train loss = 4.60555 | train err = 98.98% |
test loss = 4.60566 | test err = 99.01%
======== epoch 57 (lr: 0.00770) ========
train loss = 4.60555 | train err = 99.00% |
test loss = 4.60560 | test err = 99.01%
======== epoch 58 (lr: 0.00770) ========
train loss = 4.60575 | train err = 99.04% |
test loss = 4.60568 | test err = 99.01%
======== epoch 59 (lr: 0.00747) ========
train loss = 4.60564 | train err = 99.01% |
test loss = 4.60546 | test err = 99.01%
======== epoch 60 (lr: 0.00747) ========
train loss = 4.60566 | train err = 99.06% |
test loss = 4.60559 | test err = 99.01%
======== epoch 61 (lr: 0.00747) ========
train loss = 4.60559 | train err = 98.99% |
test loss = 4.60558 | test err = 99.01%
======== epoch 62 (lr: 0.00725) ========
train loss = 4.60551 | train err = 99.06% |
test loss = 4.60557 | test err = 99.01%
======== epoch 63 (lr: 0.00725) ========
train loss = 4.60566 | train err = 98.99% |
test loss = 4.60555 | test err = 99.01%
======== epoch 64 (lr: 0.00703) ========
train loss = 4.60563 | train err = 99.03% |
test loss = 4.60562 | test err = 99.01%
======== epoch 65 (lr: 0.00703) ========
train loss = 4.60559 | train err = 99.04% |
test loss = 4.60551 | test err = 98.94%
======== epoch 66 (lr: 0.00703) ========
train loss = 4.60555 | train err = 99.02% |
test loss = 4.60549 | test err = 99.01%
======== epoch 67 (lr: 0.00682) ========
train loss = 4.60547 | train err = 99.04% |
test loss = 4.60552 | test err = 99.01%
======== epoch 68 (lr: 0.00682) ========
train loss = 4.60565 | train err = 99.06% |
test loss = 4.60559 | test err = 99.01%
======== epoch 69 (lr: 0.00661) ========
train loss = 4.60551 | train err = 98.94% |
test loss = 4.60550 | test err = 98.94%
======== epoch 70 (lr: 0.00661) ========
train loss = 4.60546 | train err = 98.97% |
test loss = 4.60552 | test err = 99.01%
======== epoch 71 (lr: 0.00642) ========
train loss = 4.60546 | train err = 98.97% |
test loss = 4.60539 | test err = 99.01%
======== epoch 72 (lr: 0.00642) ========
train loss = 4.60555 | train err = 99.07% |
test loss = 4.60548 | test err = 99.01%
======== epoch 73 (lr: 0.00642) ========
train loss = 4.60542 | train err = 98.97% |
test loss = 4.60550 | test err = 99.01%
======== epoch 74 (lr: 0.00622) ========
train loss = 4.60550 | train err = 98.92% |
test loss = 4.60546 | test err = 99.01%
======== epoch 75 (lr: 0.00622) ========
train loss = 4.60547 | train err = 98.98% |
test loss = 4.60545 | test err = 99.01%
======== epoch 76 (lr: 0.00604) ========
train loss = 4.60542 | train err = 99.03% |
test loss = 4.60539 | test err = 99.01%
======== epoch 77 (lr: 0.00604) ========
train loss = 4.60546 | train err = 98.98% |
test loss = 4.60546 | test err = 99.01%
======== epoch 78 (lr: 0.00604) ========
train loss = 4.60552 | train err = 99.06% |
test loss = 4.60551 | test err = 99.01%
======== epoch 79 (lr: 0.00586) ========
train loss = 4.60542 | train err = 99.00% |
test loss = 4.60548 | test err = 99.01%
======== epoch 80 (lr: 0.00586) ========
train loss = 4.60553 | train err = 99.01% |
test loss = 4.60543 | test err = 99.01%
======== epoch 81 (lr: 0.00568) ========
train loss = 4.60548 | train err = 99.00% |
test loss = 4.60545 | test err = 99.01%
======== epoch 82 (lr: 0.00568) ========
train loss = 4.60537 | train err = 98.94% |
test loss = 4.60541 | test err = 99.01%
======== epoch 83 (lr: 0.00551) ========
train loss = 4.60549 | train err = 98.97% |
test loss = 4.60538 | test err = 99.01%
======== epoch 84 (lr: 0.00551) ========
train loss = 4.60545 | train err = 99.09% |
test loss = 4.60545 | test err = 99.01%
======== epoch 85 (lr: 0.00551) ========
train loss = 4.60553 | train err = 99.07% |
test loss = 4.60543 | test err = 99.01%
======== epoch 86 (lr: 0.00534) ========
train loss = 4.60541 | train err = 98.98% |
test loss = 4.60543 | test err = 99.01%
======== epoch 87 (lr: 0.00534) ========
train loss = 4.60542 | train err = 99.03% |
test loss = 4.60543 | test err = 99.01%
======== epoch 88 (lr: 0.00518) ========
train loss = 4.60545 | train err = 98.98% |
test loss = 4.60537 | test err = 99.01%
======== epoch 89 (lr: 0.00518) ========
train loss = 4.60553 | train err = 98.91% |
test loss = 4.60538 | test err = 99.01%
======== epoch 90 (lr: 0.00518) ========
train loss = 4.60535 | train err = 98.96% |
test loss = 4.60542 | test err = 99.01%
======== epoch 91 (lr: 0.00503) ========
train loss = 4.60539 | train err = 98.98% |
test loss = 4.60541 | test err = 99.01%
======== epoch 92 (lr: 0.00503) ========
train loss = 4.60538 | train err = 99.01% |
test loss = 4.60545 | test err = 99.01%
======== epoch 93 (lr: 0.00488) ========
train loss = 4.60539 | train err = 99.02% |
test loss = 4.60539 | test err = 99.01%
======== epoch 94 (lr: 0.00488) ========
train loss = 4.60539 | train err = 99.01% |
test loss = 4.60533 | test err = 99.01%
======== epoch 95 (lr: 0.00473) ========
train loss = 4.60539 | train err = 98.97% |
test loss = 4.60533 | test err = 99.01%
======== epoch 96 (lr: 0.00473) ========
train loss = 4.60532 | train err = 98.95% |
test loss = 4.60539 | test err = 99.01%
======== epoch 97 (lr: 0.00473) ========
train loss = 4.60537 | train err = 99.05% |
test loss = 4.60535 | test err = 98.94%
======== epoch 98 (lr: 0.00459) ========
train loss = 4.60541 | train err = 99.00% |
test loss = 4.60537 | test err = 99.01%
======== epoch 99 (lr: 0.00459) ========
train loss = 4.60544 | train err = 98.99% |
test loss = 4.60538 | test err = 99.01%
======== epoch 100 (lr: 0.00445) ========
train loss = 4.60533 | train err = 98.96% |
test loss = 4.60536 | test err = 99.01%
======== epoch 101 (lr: 0.00445) ========
train loss = 4.60529 | train err = 98.93% |
test loss = 4.60533 | test err = 99.01%
======== epoch 102 (lr: 0.00445) ========
train loss = 4.60531 | train err = 98.98% |
test loss = 4.60536 | test err = 99.01%
======== epoch 103 (lr: 0.00432) ========
train loss = 4.60539 | train err = 99.12% |
test loss = 4.60530 | test err = 98.94%
According to the article Swish should be input * torch.sigmoid(beta * input)
, where beta - trainable.
Your realization has no beta param.
i cant understand the test_err 99.01% on epoch 1.
Sorry
I have two quenstions.
(1)
i think it is too good for cifar100.
Why?
(2)
(When I tried to do cifar10 by modifying your source codes, the test_err was about 90% on epoch 1.
Why is cifar10_value worse than cifar100?
@katsura-jp I'm wondering why there is upsample operation in before stem ?
efficientnet-pytorch/model/efficientnet.py
Lines 153 to 156 in d9c5abd
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.