greenelab / linear_signal Goto Github PK
View Code? Open in Web Editor NEWComparing the performance of linear and nonlinear models in transcriptomic prediction
License: BSD 3-Clause "New" or "Revised" License
Comparing the performance of linear and nonlinear models in transcriptomic prediction
License: BSD 3-Clause "New" or "Revised" License
Right now labels are encoded at dataset creation time using sklearn.LabelEncoder, which sorts the labels before encoding them method one, method 2. This means that for classification where the labels are the same between datasets, the labels' encodings should also be the same. However, if there are more labels in the training set than validation set, the validation might be comparing apples to oranges.
This might be the cause of the bad performance in the all_labels
analysis, and definitely is something to make sure to get right. I'll be adding the logic to account for this (LabeledDataset.get_label_encoder()
and LabeledDataset.set_label_encoder()
) to the next PR
PytorchSupervised models should implement early stopping at some point. The current train/tune split doesn't work great though, maybe use ideas from #28 (comment) ?
There's a lot of duplicated code between PytorchSupervised and PytorchPseudolabel that can be condensed.
Likewise more of the benchmark scripts' code should be pulled into utility functions and tested
See https://github.com/greenelab/saged/blob/331a77a57564b02d8d40b1dd7b66b53a0f1b50c5/notebook/analysis/benchmark_results.ipynb tb and lupus visualizations
I cloned the repo to my local computer to run two branches' evaluations at once, and I found that the Snakefile doesn't include the logic to download and the compendium and subset it to include only the samples containing blood gene expression
It takes 7 minutes to use conda to download dependencies each time. I should probably just switch to the setup-miniconda action and enable caching.
Scaling to ~100k samples leads to a substantial slowdown in the non-model update parts of the training code. It takes awhile to subset samples, split data by study, and even pull given samples from a dataset using their IDs.
I'll need to look into how I can speed these processes up.
Ideas:
The CV splits used in the all_labels
analysis (and probably others) aren't the same between runs. I feel like this could be another set being cast to a list issue, but I'm unsure.
Code to check:
For small models ~40 percent of the runtime (~3.5 minutes) is spend just loading the subset compendium into a dataframe. This appears to be due to the slowness of the pandas.read_csv
function as opposed to a disk reading limitation, because loading the file into memory with compendium_file.readlines()
only takes 8 seconds.
Potential fixes:
Reproducible with python saged/tissue_split.py dataset_configs/smol_dataset model_configs/supervised/pytorch_supervised.yml out.txt
The code required to compare different models will have a lot of different components, so I'll list the elements here and check them off as they're implemented.
Create a base Dataset class
Implement a Dataset class for refinebio data
Implement extracting data for a given label
Implement subsetting data based on label
Implement train/test split and cross-validation, by study instead of by sample
Create a base Model class (probably using the sklearn API?)
Implement Models for skl methods
Implement Models for some semi-supervised methods
Implement non-semi-supervised versions of the same models, or separate unsupervised and supervised steps
Implement data simulation/augmentation with VAEs
Determine a way to parallelize the benchmarking process
Write the logic to take a dataset and benchmark models on it. Probably build this into the dataset object itself?
sample_split_control
and tissue_split_control
initialize the output size of the neural network models to be the same as the input size (because they were based off the imputation pretraining script).
Because neural nets are neural nets, they just learn to not use the extra 20k classes, but I should fix that issue and rerun in the future
keep_ratios.py
and other benchmark scripts require metadata files to build the RefineBioDataset objects. At some point I need to document what these files look like and which parts are actually needed to construct the dataset objects
See #37 (comment) for more
https://ui.neptune.ai/ben-heil/saged/e/SAG-4/monitoring and https://ui.neptune.ai/ben-heil/saged/e/SAG-5/monitoring correspond to the first and second fold of cross validation in all_label_comparison.py
. The second fold starts with a higher GPU memory usage, and as a result errors out.
This implies there is gpu memory being held across folds. It isn't immediately apparent why this is happening, and it doesn't fit in the scope of the current PR, so I've opened this issue.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.