Comments (5)
Hi, thanks for your interest in DL4DS
. The documentation needs some work and the creation of a tutorial is WIP. My suggestion would be to call dl4ds.SupervisedTrainer
or dl4ds.CGANTrainer
directly in your script while passing your (preprocessed) data variables. Have you tried this?
Bear in mind that the app.py
module is very experimental and is what I used to run my experiments in a cluster with a workflow manager. The data_module
is just a python script were you run your pre-processing steps (e.g., slicing data, splitting, normalizing/standardizing) and some variables are declared. These variables are called in app.py
, e.g., DATA.data_train
or DATA.predictors_train
, when feeding the training or inference steps.
from dl4ds.
Thank you for your quick response. I'm calling dl4ds.SupervisedTrainer using only data_train, data_val, data_test. But when executing trainer.run() I get:
Unexpected result of train_function (Empty logs). Please use Model.compile(..., run_eagerly=True), or tf.config.run_functions_eagerly(True) for more information of where went wrong, or file a issue/bug to tf.keras.
For what I have found, this error may be because of wrong input data shape. My input data are xr.DataArray with shape [time, latitude, longitude, 1].
from dl4ds.
The error doesn't tell me much so I'm not sure it's even related to the data (shape, format). Please provide more information about how you call the trainer and the full error.
from dl4ds.
Hi Carlos, thank you for taking care of this. Indeed, the error didn't tell much but I figured out that the problem was with the batch size, so by setting a lower batch size I was able to train a model.
I have another doubt, do all the LR data should be at the same resolution? I mean, data_train_lr, predictors_train, and static_vars should be all at the same resolution or can I have different resolutions for train_lr and static_vars for example?
from dl4ds.
Hi Andrés, I'm glad you've found the issue there. batch_size
is a tricky hyperparameter to set as it depends on many factors, such as the size of the model, the available GPU/CPU memory, the size/dimensionality of the training samples, etc. So it's very case dependant.
To answer your question: the parameters data_train_lr
, data_val_lr
and data_test_lr
require low/coarse resolution data. predictors_train
is for inputing time-varying predictors and they can come in high or intermediate resolution (DL4DS
will internally interpolate/resize the arrays when needed). static_vars
on the other hand, must be high-resolution variables, such as elevation/topography. So yes, you can have different resolutions data_train_lr
and static_vars
.
from dl4ds.
Related Issues (9)
- Error when dimension mistmatch between static variable and predictand
- static_vars = None fails for the CGANTrainer HOT 1
- requirements/environments issuse
- Problem in trainer.run() step in Colab notebook
- import dl4ds as dss: Illegal instruction (core dumped)
- batch size issue in a multi-GPU environment
- Colab example fail
- How to make predictions without the availability of future empirical data
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dl4ds.