georgemilosh / climate-learning Goto Github PK

Changing tau to -tau in the Learn2_new.py code is super easy, barely an inconvenience... but how many things would it break?

Don't recompute heavy objects that are shared between runs, like X

MCC, ConfusionMatrix, BrierScore (future) should unbias the probabilities predicted by the model accounting for the undersampling factor

I think there is no reason to have transfer learning work on u, which seems to be the case provided that load_from='last' which is default. I guess I could turn of transfer learning completely by using load_from=None. Maybe that should be defaault behavior. Or maybe default behavior should be that transfer learning occurs only with tau. Otherwise we can easily fall into a trap when not performing analysis of hyperparameter optimization carefully. I am a bit about worried about this transfer learning.

It is better to move away from print and use instead logging.log. This allows for much easier dealing with logging to different devices and different levels of verbosity

• Replace print with logging.log with proper logging levels, e.g. print(f'{X.shape = }') -> logging.info(f'{X.shape = }') or logging.level(20, f'{X.shape = }')
• Log to file and terminal
• Log important information to telegram (Alessandro knows how)

When plotting the masked mrso it seems weird: not a square grid. Understand what's going on and fix it

Make transfer learning more versatile and easier to modify from modules like resampling

Use Xarrays instead of pure netCDF4 files.
This should make it easier to use data with other format, e.g. grid resolution

There are not so many machines that can handle all the data in memory. We need to find a way to deal with this issue. There are many materials online which show how it can be achieved in tensorflow (pipelines etc). The problem is that I still get out of memory issue (the process simply gets killed).

Learn2_new.py has a method create_model.

• The ability to add weight decay (L2 regularization) is needed
• The ability to skip convolutional layers to use fully dense layers is needed

The current masked_average function is very slow and RAM intensive: design a better one

Add the possibility to have early stopping during training

Obs: this may not be very well compatible with early stopping

Make the code able to deal with multiclass (>2) classification for different intensity of heatwaves.
Things to adapt:

• label assignment
• metrics
• undersampling and its unbiasing
• ordered classes

On the first glance there seem to be no inconsistencies between the branches, thus they can be merged together. Below we list the files which many other ones depend on:

• The most difficult file to merge is Learn2_new.py
• Merging ERA_Fields_New.py is relatively easy to merge
• Merging TF_Fields.py is very easy

• Fix the location of checkpoints which seem to be outside of eons.
• Fix problems with collective: It seems when calling python resampling.py num_eons=1 we get an error
• Add more logging outputs, so that we can for example how much data actually gets inside the neural network. Also how much undersampling etc happens
• Store the selection of years (also possibly days) inside of the training folder so that we can easily check what data was used.

It would be better for fields_infos to be loaded from a json file, like we do with config.json. This is because I would like to keep the labels of the fields, while working with both CESM and Plasim. The idea would be to simply modify field_infos.json when working with CESM as opposed to Plasim

I was running
python hyperparameter_optimization.py n_trials=100
I got an error message:

File "/ClimateDynamics/MediumSpace/ClimateLearningFR/gmiloshe/CESM/models/hypop/t2-bn-dr-l2/Learn2_new.py", line 1441, in create_model
    raise ValueError(f'Invalid length for argument {arg}')
ValueError: Invalid length for argument ['relu', None]

The problem is the line that is reported does not correspond to where the error happens precisely in the code. I've noticed this before and it is quite annoying when debugging.

Perform the necessary adaptations to work with the new xarray framework

Allow to choose different portions of the data to work with. Care must be taken with the masking procedure

Right now the latitude used for regularization is hardcoded. This is bad for generalization. We need to get it from the fields passing through all the functions up to where it needs to go.

A possibility is to add a tunnel argument that is simply passed through all the functions in the main Learn2_new code that helps for inheritance

Keep track of the runs in a json file instead of txt one. This way we can save metadata about the run like when it started, when (if) it ended, etc.

Put general purpose routines inside a git submodule so it can be accessed from other repos easily.

This may break backward compatibility

Also 'u' is multiply defined in the config.json

In the context of 1000 year long dataset it seemed that global zg500 mattered less, therefore decision was taken to ignore this more advanced treatment. In 8000 year long dataset (possibly also due to weaker soil atmosphere interactions) global zg500 is more prominent, also possibly for shorter heat waves this could also be the case. In Besombes at al 2021 we see this setup for their generative model. VAE trained on Plasim shows that it is harder to generate continuous outputs with standard padding (convolution), although deeper network seems to perform better in this regard.

• Take into account the connectivity
• Take into account that grid points are not equally spaced

Note that https://www.tensorflow.org/install/pip#linux tensorflow tutorial now recommends installing with pip.

It is not even possible to install tensorflow-gpu anymore on IPSL cluster.

Choose labels which correspond to the first few days of the onset of heat waves and see what happens. Currently we mix the labels of heat waves which correspond to already started heat waves (that take 14 days) and heat waves that have been on-going for a long time (and are still above 5 %). This potentially destroys dynamical information and prevents us from extracting it from the DNN

• Determine what are the characteristics of the events we want to keep as class heat wave onset and the amount of positive labels we thus keep
• Possibly introduce a class for "on-going heat wave"

Redesign the region-wise masks to work with xarray

In each run folder create the files area_integral.npy and A.npy

Cleanup old features?

• Is rolling still necessary now that we have wrap around longitude?
• Is it worth saving longitude and latitude in prepare_XY

Further optimization

• Save data with already monotonized years, so we don't need to do it run-time

Create a new version of ERA_Fields_New.Plasim_Field that uses xarray

Implement early stopping for eons in resampling.py: meaning that if the score doesn't improve over the eons we stop training

Move all general purpose functions/classes to a utilities.py module

• kwargs support
• modularity
• json file configs
• k_fold_cross_val implemented
• trainer class
• reconstruction.py should not load all the years
• classification.py with a possible choice of classifiers and hyperparameters

This will allow us to selectively load different subroutines of Learn2.py. Sometimes when analysing the predictions of the model we don't need to load all the data. This leads to unnecessary delays, in particular when working with multiple tau (lag times)

• Create modular support for PLASIM/Learn2.py similar to the one in VAE/vae_learn1.py
• The idea is to also be able to load specific permutation of years from the *.nc file without loading the who *.nc file
• add tau as argument of PLASIM/Learn2.py
• config file?
• Include Brier score in the metric evaluation (subclass tensorflow)
• Write an analogue for PLASIM/Recalc_Tau_Metrics.py routine that would benefit from these changes

• manage the eon folders correctly

Split Inspect network

The point is to be able to load the fields from the CESM model using the same classes we have written

Currently we are mixing years which start very cold and very warm in summer. Probably it is easier and more interesting to predict if there is 14 day anomaly above this level

Allow a better control on from where to do transfer learning

George has to fix the error which curiously occurs when passing from vae_learn1.py to vae_learn2.py

"Traceback (most recent call last):
File "/ClimateDynamics/MediumSpace/ClimateLearningFR/gmiloshe/VAE/vae_learn2.py", line 274, in
X, LON, LAT, vae, Z_DIM, N_EPOCHS, INITIAL_EPOCH, BATCH_SIZE, LEARNING_RATE, checkpoint_path, checkpoint_name, myinput, history = PrepareDataAndVAE()
File "/ClimateDynamics/MediumSpace/ClimateLearningFR/gmiloshe/VAE/vae_learn2.py", line 259, in PrepareDataAndVAE
vae, history, N_EPOCHS, INITIAL_EPOCH, checkpoint, checkpoint_path = ConstructVAE(INPUT_DIM, Z_DIM, checkpoint_name, N_EPOCHS, myinput, K1, K2, from_logits=False, mask_weights=filter_mask)
File "/ClimateDynamics/MediumSpace/ClimateLearningFR/gmiloshe/VAE/vae_learn2.py", line 169, in ConstructVAE
decoder_input, decoder_output, decoder = tff.build_decoder_skip(input_dim = 64,
File "/ClimateDynamics/MediumSpace/ClimateLearningFR/gmiloshe/ERA/TF_Fields.py", line 287, in build_decoder_skip
decoder_outputs = ConstMul(mask,(~mask)*0.5)(x[-1]) # This will multiply the input by mask consisting of 0's (False) and 1's (True). Because the decoder is expected to reconstruct sigmoid function we add 0.5 where there were 0's
File "/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer.py", line 951, in call
return self._functional_construction_call(inputs, args, kwargs,
File "/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1090, in _functional_construction_call
outputs = self._keras_tensor_symbolic_call(
File "/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer.py", line 822, in _keras_tensor_symbolic_call
return self._infer_output_signature(inputs, args, kwargs, input_masks)
File "/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/keras/engine/base_layer.py", line 863, in _infer_output_signature
outputs = call_fn(inputs, *args, **kwargs)
File "/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py", line 670, in wrapper
raise e.ag_error_metadata.to_exception(e)
NotImplementedError: in user code:

/ClimateDynamics/MediumSpace/ClimateLearningFR/gmiloshe/ERA/TF_Fields.py:49 call  *
    return inputs * self.const_a + self.const_b
/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/ops/math_ops.py:1175 binary_op_wrapper
    out = r_op(x)
/home/gmiloshe/miniconda3/envs/myenvnew/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:852 __array__
    raise NotImplementedError(

NotImplementedError: Cannot convert a symbolic Tensor (Placeholder:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

"