greenelab / linear_signal Goto Github PK

Right now labels are encoded at dataset creation time using sklearn.LabelEncoder, which sorts the labels before encoding them method one, method 2. This means that for classification where the labels are the same between datasets, the labels' encodings should also be the same. However, if there are more labels in the training set than validation set, the validation might be comparing apples to oranges.

This might be the cause of the bad performance in the all_labels analysis, and definitely is something to make sure to get right. I'll be adding the logic to account for this (LabeledDataset.get_label_encoder() and LabeledDataset.set_label_encoder()) to the next PR

PytorchSupervised models should implement early stopping at some point. The current train/tune split doesn't work great though, maybe use ideas from #28 (comment) ?

There's a lot of duplicated code between PytorchSupervised and PytorchPseudolabel that can be condensed.

Likewise more of the benchmark scripts' code should be pulled into utility functions and tested

See https://github.com/greenelab/saged/blob/331a77a57564b02d8d40b1dd7b66b53a0f1b50c5/notebook/analysis/benchmark_results.ipynb tb and lupus visualizations

I cloned the repo to my local computer to run two branches' evaluations at once, and I found that the Snakefile doesn't include the logic to download and the compendium and subset it to include only the samples containing blood gene expression

It takes 7 minutes to use conda to download dependencies each time. I should probably just switch to the setup-miniconda action and enable caching.

Scaling to ~100k samples leads to a substantial slowdown in the non-model update parts of the training code. It takes awhile to subset samples, split data by study, and even pull given samples from a dataset using their IDs.

I'll need to look into how I can speed these processes up.
Ideas:

• Use less data (boooo)
• Presplit data into train/test sets with different seeds. Expensive(ish) in terms of disk space, but should be much faster
• Look into why index access into a 100k element dataframe is so slow. Is it a limitation of pandas itself, or am I doing something inefficient (like is my access pattern correct in the light of pandas using column major order)

The CV splits used in the all_labels analysis (and probably others) aren't the same between runs. I feel like this could be another set being cast to a list issue, but I'm unsure.

Code to check:

• get_cv_splits
• dataset constructors

More documentation than issue, here are the sample counts and study counts for the manually labeled samples in recount3

Samples per tissue:

Unique studies per tissue:

For small models ~40 percent of the runtime (~3.5 minutes) is spend just loading the subset compendium into a dataframe. This appears to be due to the slowness of the pandas.read_csv function as opposed to a disk reading limitation, because loading the file into memory with compendium_file.readlines() only takes 8 seconds.

Potential fixes:

• Pickle the dataframe and load it
• Load the file into a numpy array and use the numpy constructor for the dataframe
• Try dask again for parallelism since we appear to be CPU bound instead of disk bound

Reproducible with python saged/tissue_split.py dataset_configs/smol_dataset model_configs/supervised/pytorch_supervised.yml out.txt

The code required to compare different models will have a lot of different components, so I'll list the elements here and check them off as they're implemented.

• Create a base Dataset class

• Implement a Dataset class for refinebio data

• Implement extracting data for a given label

• Implement subsetting data based on label

• Implement train/test split and cross-validation, by study instead of by sample

• Create a base Model class (probably using the sklearn API?)

• Implement Models for skl methods

• Implement Models for some semi-supervised methods

• Implement non-semi-supervised versions of the same models, or separate unsupervised and supervised steps

• Implement data simulation/augmentation with VAEs

• Determine a way to parallelize the benchmarking process

• Write the logic to take a dataset and benchmark models on it. Probably build this into the dataset object itself?

sample_split_control and tissue_split_control initialize the output size of the neural network models to be the same as the input size (because they were based off the imputation pretraining script).

Because neural nets are neural nets, they just learn to not use the extra 20k classes, but I should fix that issue and rerun in the future

keep_ratios.py and other benchmark scripts require metadata files to build the RefineBioDataset objects. At some point I need to document what these files look like and which parts are actually needed to construct the dataset objects

See #37 (comment) for more

https://ui.neptune.ai/ben-heil/saged/e/SAG-4/monitoring and https://ui.neptune.ai/ben-heil/saged/e/SAG-5/monitoring correspond to the first and second fold of cross validation in all_label_comparison.py. The second fold starts with a higher GPU memory usage, and as a result errors out.

This implies there is gpu memory being held across folds. It isn't immediately apparent why this is happening, and it doesn't fit in the scope of the current PR, so I've opened this issue.

Here's the distribution of the number of genes in the canonicalpathways dataset from PLIER.

Obtained via

library(PLIER)
data(canonicalPathways)
hist(colSums(canonicalPathways), breaks=30)