gzhuoyan / multimodal-remote-sensing-toolkit

A python tool to perform deep learning experiments on multimodal remote sensing data.

Deep Learning Methods for Multi-modal Remote Sensing Classification

MMRS is a python tool to perform deep learning experiments on multi-modal remote sensing data.

This repository is developed on the top of DeepHyperX .

Models

Currently, the following deep learning methods are available:

• Two-Branch CNN
• EndNet
• MDL-Hong
• FusAtNet
• S2ENet (ours)

Datasets

• Houston2013 (Hyperspectral and LiDAR Data): The processed .mat files can be obtained on google drive.

• Trento Data (Hyperspectral and LiDAR Data): Trento dataset is provided by Professor Prof. L. Bruzzone from the University of Trento.

Quickstart using Colab

You can use MMRS on Google Colab Notebook without any installation. You can run all cells without any modifications to see how everything works.

Usage

Start a Visdom server: python -m visdom.server and go to http://localhost:8097 to see the visualizations.

Then, run the script main.py.

The most useful arguments are:

• --model to specify the model (e.g. 'S2ENet', 'Middle_fusion_CNN'),
• --dataset to specify which dataset to use (e.g. 'Houston2013', 'Trento'),
• the --cuda switch to run the neural nets on GPU. The tool fallbacks on CPU if this switch is not specified.

There are more parameters that can be used to control more finely the behaviour of the tool. See python main.py -h for more information.

Examples:

!python main.py --model S2ENet --flip_augmentation --patch_size 7 --epoch 128 --lr 0.001 --batch_size 64 --seed 0 --dataset Houston2013 --folder '../' --train_set '../Houston2013/TRLabel.mat' --test_set '../Houston2013/TSLabel.mat' --cuda 0

For more features please refer to DeepHyperX.

Citation

If you find this work valuable or use our code in your own research, please consider citing us:

S. Fang, K. Li and Z. Li, "S²ENet: Spatial–Spectral Cross-Modal Enhancement Network for Classification of Hyperspectral and LiDAR Data," in IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1-5, 2022, Art no. 6504205, doi: 10.1109/LGRS.2021.3121028.

Bibtex format :

@ARTICLE{9583936, author={Fang, Sheng and Li, Kaiyu and Li, Zhe}, journal={IEEE Geoscience and Remote Sensing Letters}, title={S²ENet: Spatial–Spectral Cross-Modal Enhancement Network for Classification of Hyperspectral and LiDAR Data}, year={2022}, volume={19}, number={}, pages={1-5}, doi={10.1109/LGRS.2021.3121028}}