abadcandy / baidubigdata19-urfc

my solution with 0.67 accuracy

BaiDuBigData19-URFC

my two networks solution with 0.67 accuracy for 9 classification.

主要为了用Pytorch复现 https://github.com/czczup/UrbanRegionFunctionClassification 这位大神的tensorflow实现的双分支网络baseline， 同时visit数据的转换和链接中visit2array.py效果一致，即转为7×26×24(天×周×小时)的特征矩阵。

不同点：

1. 数据预处理：除了简单的平移旋转降噪外还事先去除了全黑和全白图片、去雾、直方图均衡化等，剩余训练集图片数为39730张。
2. 图片网络：采用原始3×100×100的尺寸输入，利用imagenet的预训练模型se_resnext101_32x4d进行微调。
3. visit网络：输入尺寸为7×26×24，不过与tf版本不同之处为前者将24作为通道数，该版本将7作为通道数，这样长宽基本一致，可以利用cifar10或cifar100 的预训练ResNet系列模型进行微调，这里采用的是无预训练的dpn26网络。
4. 特征融合：图片网络最后一层的特征向量维度为256，visit网络最后特征维度为64，concat后为320，最后接9个节点的全连接层进行分类。

使用说明：

1. 预处理数据下载链接：https://pan.baidu.com/s/1UxcvfsGyIZ1kGGcKmvuj8A 提取码：r3w4

在当前目录下新建data文件夹，将下载好的数据解压至该目录下，最后可以看到data文件夹下有npy,train.test三个子文件夹， 其中npy里存有转换好的train_visit和test_visit，train和test两个子文件夹里分别存放了筛选和预处理后的39730张训练图片，以及原始的1w张测试图片。

2. 执行 pip install -r requirements.txt 安装必要的运行库。

3. 执行 python multimain.py 即可开始训练和测试，其中一些超参数如epoch,batch_size等可在config.py中修改。

4. 等第3步执行完后会在sumbit文件夹下生成csv格式的预测结果，为了与提交系统要求保持一致需要再运行 python submission.py，最终在submit文件夹下得到submit.txt即可提交。

提交结果如图：

baidubigdata19-urfc's Issues

请问有预处理相关的文件吗

中断后继续训练

你好，想请教您训练过程中暂定后如何继续训练。
Traceback (most recent call last):
File "multimain.py", line 249, in
main()
File "multimain.py", line 178, in main
checkpoint = torch.load(r'./checkpoints/best_models/seresnext101_dpn92_defrog_multimodal_fold_0_model_best_loss.pth.tar')
File "/home/moer/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 366, in load
f = open(f, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: './checkpoints/best_models/seresnext101_dpn92_defrog_multimodal_fold_0_model_best_loss.pth.tar'

模型下载错误

----------------------------------------------- [START 2019-10-29 19:04:31] ---------------------------------------------------

                       |------------ Train -------|----------- Valid ---------|----------Best Results---|------------|

mode iter epoch | acc loss f1_macro | acc loss f1_macro | loss f1_macro | time |
-------------------------------------------------------------------------------------------------------------------------|
Downloading: "http://data.lip6.fr/cadene/pretrainedmodels/se_resnext101_32x4d-3b2fe3d8.pth" to /Users/didi/.cache/torch/checkpoints/se_resnext101_32x4d-3b2fe3d8.pth
5%|▍ | 9289184/196466866 [01:32<31:04, 100393.23it/s]
Traceback (most recent call last):
File "/Users/didi/Desktop/git/BaiDuBigData19-URFC/multimain.py", line 249, in
main()
File "/Users/didi/Desktop/git/BaiDuBigData19-URFC/multimain.py", line 164, in main
model=MultiModalNet("se_resnext101_32x4d","dpn26",0.5)
File "/Users/didi/Desktop/git/BaiDuBigData19-URFC/multimodal.py", line 248, in init
img_model = pretrainedmodels.dict[backbone1](num_classes=1000, pretrained='imagenet') #seresnext101
File "/anaconda3/lib/python3.7/site-packages/pretrainedmodels/models/senet.py", line 441, in se_resnext101_32x4d
initialize_pretrained_model(model, num_classes, settings)
File "/anaconda3/lib/python3.7/site-packages/pretrainedmodels/models/senet.py", line 373, in initialize_pretrained_model
model.load_state_dict(model_zoo.load_url(settings['url']))
File "/anaconda3/lib/python3.7/site-packages/torch/hub.py", line 433, in load_state_dict_from_url
_download_url_to_file(url, cached_file, hash_prefix, progress=progress)
File "/anaconda3/lib/python3.7/site-packages/torch/hub.py", line 377, in _download_url_to_file
.format(hash_prefix, digest))
RuntimeError: invalid hash value (expected "3b2fe3d8", got "200fdf586d5c96a4798edb0700a486d14fc0578cf94d91ebe44af17c89853475")

运行环境

报错的库imgaug==0.2.8，torch==1.1.0，我使用的是3.7，运行的时候，不支持。请问您的具体的运行环境？

具体报错如下：
Traceback (most recent call last):
File "multimain.py", line 249, in
main()
File "multimain.py", line 212, in main
train_metrics = train(train_loader,model,criterion,optimizer,epoch,val_metrics,best_results,start)
File "multimain.py", line 56, in train
for i,(images,visit,target) in enumerate(train_loader):
File "D:\computer_language\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 582, in next
return self._process_next_batch(batch)
File "D:\computer_language\anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
AttributeError: Traceback (most recent call last):
File "D:\computer_language\anaconda3\lib\site-packages\torch\utils\data_utils\worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "D:\computer_language\anaconda3\lib\site-packages\torch\utils\data_utils\worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "E:\大数据\2019大数据竞赛初赛数据集\BaiDuBigData19-URFC-master\multimodal.py", line 62, in getitem
X = self.augumentor(X)
File "E:\大数据\2019大数据竞赛初赛数据集\BaiDuBigData19-URFC-master\multimodal.py", line 99, in augumentor
image_aug = augment_img.augment_image(image)
File "D:\computer_language\anaconda3\lib\site-packages\imgaug\augmenters\meta.py", line 405, in augment_image
ia.do_assert(image.ndim in [2, 3],
AttributeError: 'NoneType' object has no attribute 'ndim'

链接失效了

您好~ 这个百度云盘链接失效了。能否再附一个分享链接呢？ 感谢：）

My Computer went dead.

help !!!!
thinks for your kernnal, but...
My computer has a gpu GTX950m with a 4g memory, with the batchsize of 8(16+ dead in the beginning),when I finished the first epoch, it did not respond............

and the training errror was less than 0.5...
I am hungry...

没有参数属性

AttributeError: 'collections.OrderedDict' object has no attribute 'parameters'

运行报错

Traceback (most recent call last):
File "multimain.py", line 249, in
main()
File "multimain.py", line 212, in main
train_metrics = train(train_loader,model,criterion,optimizer,epoch,val_metrics,best_results,start)
File "multimain.py", line 62, in train
output = model(images,visit)
File "/home/csu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/media/csu/新加卷/AI Competition/Competition/一带一路国际大数据竞赛/初赛赛题/方案二/multimodal.py", line 275, in forward
x_vis=self.visit_model(x_vis)
File "/home/csu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/media/csu/新加卷/AI Competition/Competition/一带一路国际大数据竞赛/初赛赛题/方案二/multimodal.py", line 218, in forward
out = F.avg_pool2d(out, 4)
RuntimeError: Given input size: (2432x4x3). Calculated output size: (2432x1x0). Output size is too small at /pytorch/aten/src/THCUNN/generic/SpatialAveragePooling.cu:63
This problem has confused me for a few days,I hope you can offer me a solution to this problem,thanks a lot.

Main error

I get error like this when run multimain:
TypeError: new() received an invalid combination of arguments - got (float, int, int, int), but expected one of:

• (torch.device device)
• (torch.Storage storage)
• (Tensor other)
• (tuple of ints size, torch.device device)
• (object data, torch.device device)

My pthon version is 2.7 and torch is 1.1..0. And I don't know why? Can you answer it, and thank you very much.

请问测试集的标签为什么全是0

help

大神，我比较关注数据预处理部分，请问预处理（去全黑、全白等）部分在哪个代码文件里呀？