neptune-ai / open-solution-salt-identification

Open solution to the TGS Salt Identification Challenge

TGS Salt Identification Challenge

This is an open solution to the TGS Salt Identification Challenge.

Note

Unfortunately, we can no longer provide support for this repo. Hopefully, it should still work, but if it doesn't, we cannot really help.

More competitions 🎇

Check collection of public projects 🎁, where you can find multiple Kaggle competitions with code, experiments and outputs.

Our goals

We are building entirely open solution to this competition. Specifically:

1. Learning from the process - updates about new ideas, code and experiments is the best way to learn data science. Our activity is especially useful for people who wants to enter the competition, but lack appropriate experience.
2. Encourage more Kagglers to start working on this competition.
3. Deliver open source solution with no strings attached. Code is available on our GitHub repository 💻. This solution should establish solid benchmark, as well as provide good base for your custom ideas and experiments. We care about clean code 😃
4. We are opening our experiments as well: everybody can have live preview on our experiments, parameters, code, etc. Check: TGS Salt Identification Challenge 📈 or screen below.

Train and validation monitor 📊

Disclaimer

In this open source solution you will find references to the neptune.ai. It is free platform for community Users, which we use daily to keep track of our experiments. Please note that using neptune.ai is not necessary to proceed with this solution. You may run it as plain Python script 🐍.

How to start?

Learn about our solutions

1. Check Kaggle forum and participate in the discussions.
2. See solutions below:

Link to Experiments	CV	LB	Open
solution 1	0.413	0.745	True
solution 2	0.794	0.798	True
solution 3	0.807	0.801	True
solution 4	0.802	0.809	True
solution 5	0.804	0.813	True
solution 6	0.819	0.824	True
solution 7	0.829	0.837	True
solution 8	0.830	0.845	True
solution 9	0.853	0.849	True

Start experimenting with ready-to-use code

You can jump start your participation in the competition by using our starter pack. Installation instruction below will guide you through the setup.

Installation

Clone repository

git clone https://github.com/minerva-ml/open-solution-salt-identification.git

Set-up environment

You can setup the project with default env variables and open NEPTUNE_API_TOKEN by running:

source Makefile

I suggest at least reading the step-by-step instructions to know what is happening.

Install conda environment salt

conda env create -f environment.yml

After it is installed you can activate/deactivate it by running:

conda activate salt

conda deactivate

Register to the neptune.ai (if you wish to use it) even if you don't register you can still see your experiment in Neptune. Just go to shared/showroom project and find it.

Set environment variables NEPTUNE_API_TOKEN and CONFIG_PATH.

If you are using the default neptune.yaml config then run:

export export CONFIG_PATH=neptune.yaml

otherwise you can change to your config.

Registered in Neptune:

Set NEPTUNE_API_TOKEN variable with your personal token:

export NEPTUNE_API_TOKEN=your_account_token

Create new project in Neptune and go to your config file (neptune.yaml) and change project name:

project: USER_NAME/PROJECT_NAME

Not registered in Neptune:

open token

export NEPTUNE_API_TOKEN=eyJhcGlfYWRkcmVzcyI6Imh0dHBzOi8vdWkubmVwdHVuZS5tbCIsImFwaV9rZXkiOiJiNzA2YmM4Zi03NmY5LTRjMmUtOTM5ZC00YmEwMzZmOTMyZTQifQ==

Create data folder structure and set data paths in your config file (neptune.yaml)

Suggested directory structure:

project
|--   README.md
|-- ...
|-- data
    |-- raw
         |-- train 
            |-- images 
            |-- masks
         |-- test 
            |-- images
         |-- train.csv
         |-- sample_submission.csv
    |-- meta
        │-- depths.csv
        │-- metadata.csv # this is generated
        │-- auxiliary_metadata.csv # this is generated
    |-- stacking_data
        |-- out_of_folds_predictions # put oof predictions for multiple models/pipelines here
    |-- experiments
        |-- baseline # this is where your experiment files will be dumped
            |-- checkpoints # neural network checkpoints
            |-- transformers # serialized transformers after fitting
            |-- outputs # outputs of transformers if you specified save_output=True anywhere
            |-- out_of_fold_train_predictions.pkl # oof predictions on train
            |-- out_of_fold_test_predictions.pkl # oof predictions on test
            |-- submission.csv
        |-- empty_non_empty 
        |-- new_idea_exp 

in neptune.yaml config file change data paths if you decide on a different structure:

  # Data Paths
  train_images_dir: data/raw/train
  test_images_dir: data/raw/test
  metadata_filepath: data/meta/metadata.csv
  depths_filepath: data/meta/depths.csv
  auxiliary_metadata_filepath: data/meta/auxiliary_metadata.csv
  stacking_data_dir: data/stacking_data

Run experiment based on U-Net:

Prepare metadata:

python prepare_metadata.py

Training and inference. Everything happens in main.py. Whenever you try new idea make sure to change the name of the experiment:

EXPERIMENT_NAME = 'baseline'

to a new name.

python main.py

You can always change the pipeline you want ot run in the main. For example, if I want to run just training and evaluation I can change `main.py':

if __name__ == '__main__':
    train_evaluate_cv()

References

1.Lovash Loss

@InProceedings{Berman_2018_CVPR,
author = {Berman, Maxim and Rannen Triki, Amal and Blaschko, Matthew B.},
title = {The Lovász-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2018}
}

Get involved

You are welcome to contribute your code and ideas to this open solution. To get started:

1. Check competition project on GitHub to see what we are working on right now.
2. Express your interest in paticular task by writing comment in this task, or by creating new one with your fresh idea.
3. We will get back to you quickly in order to start working together.
4. Check CONTRIBUTING for some more information.

User support

There are several ways to seek help:

1. Kaggle discussion is our primary way of communication.
2. Submit an issue directly in this repo.

open-solution-salt-identification's Issues

Implement easy fine-tuning handling

Train empty/not empty model

Could be useful for postprocessong

introduce depth to the model

augmentation is non-trivial

experiment with cumsum images

explore augmentation options

GPU utilization is 0 most of the time

nvidia-smi gives 0% percentage GPU-Util most of the time(in offline mode). Am I doing something wrong? Using the same config file as in [solution-3]. CPU usage is high most of the time.

How to transfer learn greyscale images

train multistage

• freeze resnet
• unfreeze

Implement mean teacher semi-supervised trainer

Fine tune with frozen batch norm

Train second level model

Train second level model predicting iout ler image. Perheps drop if low score.

train with sgd

setup k-fold validation

Implement attention into decoders

Implement post-processing block JOSE

implement center_encoder l2 loss

Supposedly adding l2 loss on bottom-center block leads to better training->accuracy

implemented hand-crafted architecture

The structure of the data is a bit different from your typical image problem:

• depth can be used in the model
• the image is build top-to-botton (that is why cumsum is often used)

train_evaluate_cv fails due to out_of_memory

TypeError: can't multiply sequence by non-int of type 'float' while training

I'm trying to run locally solution-3

Full stack of the error:

TypeError                                 Traceback (most recent call last)
<ipython-input-5-2da0ffaf5447> in <module>()
----> 1 train()

<ipython-input-3-0a8b44c1d62a> in train()
    314     pipeline_network = unet(config=CONFIG, train_mode=True)
    315     pipeline_network.clean_cache()
--> 316     pipeline_network.fit_transform(data)
    317     pipeline_network.clean_cache()
    318 

~/anaconda3/lib/python3.6/site-packages/steppy/base.py in fit_transform(self, data)
    321             else:
    322                 step_inputs = self._unpack(step_inputs)
--> 323             step_output_data = self._cached_fit_transform(step_inputs)
    324         return step_output_data
    325 

~/anaconda3/lib/python3.6/site-packages/steppy/base.py in _cached_fit_transform(self, step_inputs)
    441             else:
    442                 logger.info('Step {}, fitting and transforming...'.format(self.name))
--> 443                 step_output_data = self.transformer.fit_transform(**step_inputs)
    444                 logger.info('Step {}, persisting transformer to the {}'
    445                             .format(self.name, self.exp_dir_transformers_step))

~/anaconda3/lib/python3.6/site-packages/steppy/base.py in fit_transform(self, *args, **kwargs)
    603             dict: outputs
    604         """
--> 605         self.fit(*args, **kwargs)
    606         return self.transform(*args, **kwargs)
    607 

~/Desktop/ml/salt/open-solution-salt-identification-master/common_blocks/models.py in fit(self, datagen, validation_datagen, meta_valid)
     61             self.model = self.model.cuda()
     62 
---> 63         self.callbacks.set_params(self, validation_datagen=validation_datagen, meta_valid=meta_valid)
     64         self.callbacks.on_train_begin()
     65 

~/Desktop/ml/salt/open-solution-salt-identification-master/common_blocks/callbacks.py in set_params(self, *args, **kwargs)
     89     def set_params(self, *args, **kwargs):
     90         for callback in self.callbacks:
---> 91             callback.set_params(*args, **kwargs)
     92 
     93     def on_train_begin(self, *args, **kwargs):

~/Desktop/ml/salt/open-solution-salt-identification-master/common_blocks/callbacks.py in set_params(self, transformer, validation_datagen, *args, **kwargs)
    235         self.optimizer = transformer.optimizer
    236         self.loss_function = transformer.loss_function
--> 237         self.lr_scheduler = ExponentialLR(self.optimizer, self.gamma, last_epoch=-1)
    238 
    239     def on_train_begin(self, *args, **kwargs):

~/anaconda3/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, gamma, last_epoch)
    155     def __init__(self, optimizer, gamma, last_epoch=-1):
    156         self.gamma = gamma
--> 157         super(ExponentialLR, self).__init__(optimizer, last_epoch)
    158 
    159     def get_lr(self):

~/anaconda3/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in __init__(self, optimizer, last_epoch)
     19                                    "in param_groups[{}] when resuming an optimizer".format(i))
     20         self.base_lrs = list(map(lambda group: group['initial_lr'], optimizer.param_groups))
---> 21         self.step(last_epoch + 1)
     22         self.last_epoch = last_epoch
     23 

~/anaconda3/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in step(self, epoch)
     29             epoch = self.last_epoch + 1
     30         self.last_epoch = epoch
---> 31         for param_group, lr in zip(self.optimizer.param_groups, self.get_lr()):
     32             param_group['lr'] = lr
     33 

~/anaconda3/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in get_lr(self)
    159     def get_lr(self):
    160         return [base_lr * self.gamma ** self.last_epoch
--> 161                 for base_lr in self.base_lrs]
    162 
    163 

~/anaconda3/lib/python3.6/site-packages/torch/optim/lr_scheduler.py in <listcomp>(.0)
    159     def get_lr(self):
    160         return [base_lr * self.gamma ** self.last_epoch
--> 161                 for base_lr in self.base_lrs]
    162 
    163 

TypeError: can't multiply sequence by non-int of type 'float'

SO suggest to convert list to numpy this way np.asarray(coff) * C

• but actually I'm kind confused where to apply it

Implement advanced optimization techniques

• cyclic learning rate
• warm restarts

train on 128 + 32x2 pad

Problem with submission.csv

There is problem with submission.csv file as kaggle is not accepting the file and showing error

Evaluation Exception: Index was outside the bounds of the array.

There might be some problem with rle encoding

experiment with CNN1d + RNN architecture

CNN1d can go both row-wise and column-wise both should be checked but column-wise should be checked first

Drop strictly vertical targets

Some target masks ate strictly vertical . drop them during training

Can not find the 'metadata.csv'

When I run the command 'python main.py -- train--pipeline_name unet', I got the error "Can not find the 'metadata.csv'" . I checked all the files of salt-detection and still can not find it. Did I miss something important?

Explore using cumsum as another channel

Confusion Matrix

As per the discussions on Kaggle, yours implementation is the only implementation that is fully correct for the given metric but there is one thing that I couldn't understand as per your code. Here are these three functions:

def compute_ious(gt, predictions):
    gt_ = get_segmentations(gt)
    predictions_ = get_segmentations(predictions)

    if len(gt_) == 0 and len(predictions_) == 0:
        return np.ones((1, 1))
    elif len(gt_) != 0 and len(predictions_) == 0:
        return np.zeros((1, 1))
    else:
        iscrowd = [0 for _ in predictions_]
        ious = cocomask.iou(gt_, predictions_, iscrowd)
        if not np.array(ious).size:
            ious = np.zeros((1, 1))
        return ious


def compute_precision_at(ious, threshold):
    mx1 = np.max(ious, axis=0)
    mx2 = np.max(ious, axis=1)
    tp = np.sum(mx2 >= threshold)
    fp = np.sum(mx2 < threshold)
    fn = np.sum(mx1 < threshold)
    return float(tp) / (tp + fp + fn)

def compute_eval_metric(gt, predictions):
    thresholds = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]
    ious = compute_ious(gt, predictions)
    precisions = [compute_precision_at(ious, th) for th in thresholds]
    return sum(precisions) / len(precisions)

Now, given the fact that compute_ious function works on a single prediction and it's corresponding groundtruth, ious will be a singleton array. Then, how are you calculating TP/FP from that? Am I missing something here?

try hypercolumn approach

https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/64645

Explore gabor filters representation

http://nbviewer.jupyter.org/github/bicv/LogGabor/blob/master/LogGabor.ipynb

Something's wrong in multiclass dice loss

Here https://github.com/neptune-ml/open-solution-salt-detection/blob/master/src/models.py#L270 you write that target is a tensor [NxHxW] and later, here: https://github.com/neptune-ml/open-solution-salt-detection/blob/master/src/models.py#L291 you treat target as if it was a 4D tensor. The docstring is wrong or implementation is wrong.

train unet from scratch

what's the reason of the cumsum operation?

thanks for sharing!
why did you perform cumsum on the image pixels?

Test whether the last pixel to be the right, bottom of the image is allowed

Implement the lovasz softmax loss

Implement SWA optimization

https://towardsdatascience.com/stochastic-weight-averaging-a-new-way-to-get-state-of-the-art-results-in-deep-learning-c639ccf36a

https://github.com/timgaripov/swa

"Land Cover Classification from Satellite Imagery With U-Net and Lovasz-Softmax Loss" - Alexander Rakhlin, CVPR 2018

validation

Experiment with random gradient

Increase unet resolution

https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/64645

Implement feature pyramid with global attention

https://arxiv.org/pdf/1805.10180.pdf

dataset is small

implement batch norm in the decoder part

KFoldBySortedValue issue

Hello neptune-team, I'm a fellow Kaggler. I find your contribution very valuable and I'm exploring your code.

I stumbled on KFoldBySortedValue while doing static analysis.

Shouldn't

https://github.com/neptune-ml/open-solution-salt-detection/blob/239016c742d913a6822d33edb6f3beaac959cc9b/src/utils.py#L416

be

def get_n_splits(self, X=None, y=None, groups=None):
        return self.n_splits

Btw... did you get any improvement in sorting by depth the dataset?

Thank you

tta decreases results

experiment with adding depth channel to the image

add normalized depth channel where first row has all values equal to 0 and 101 row has all values equal to 1. Potentially one more channel where we multiply pixel value and normalized depth is a good idea too.

physics of seismic data

Cannot find the 'metadata.csv'

After run the code, I'm sorry to find 'metadata.csv' does not exist. And I could not find it in the homepage of TGS. I was wondering how to solve this problem.

implement y-stretch augmentation

• stretch in y direction + cut + rescale to the original size
• it should be possible with iaa.Scale fro imgaug

train with encoder LR smaller

impelment nms for joining predictions from different models(folds)