openbioml / bio-chem-lm

High level goals

This repository is dedicated specifically to the development of (Large) Language Models, and/or Language/Structure models in the bio-chem space.

Further details can be found here

Bio-LM PubChem Selfies

We are training an Electra-style model on the PubChem dataset with SELFIES representations. The SELFIES is a chemical language that is based on the SMILES language, but is more robust. More info about SELFIES can be found here.

We have released the dataset to HuggingFace Datasets, which contains ~110M compounds in total.

We will perform a hyperparameter search using Maximal Update Parameterization to find a good set of hyperparameters to transfer to a larger model. To launch a sweep on the cluster, run

sbatch --array=1-N mup_train.sh

bio-chem-lm's Issues

Add BACE dataset to HuggingFace

Dataset found in DeepChem, let's see if we can add it to Huggingface so we can

• Add SELIFES
• Make it easy to use for finetuning

Add Tox21 Dataset to HuggingFace

Found here: https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#tox21-datasets

Only need SR-p53 task

• Add selfies column

Add Delaney Dataset to HuggingFace

Found here: https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#delaney-datasets

• Add selfies column

Add PCBA Dataset to HuggingFace

Add dataset from this paper: https://par.nsf.gov/servlets/purl/10168888

Need to extract right part from DeepChem: https://github.com/deepchem/deepchem/blob/master/deepchem/molnet/load_function/pcba_datasets.py

We select one
of the largest datasets, the dataset with ID 686978 among the 128
datasets to evaluate our method. PCBA-93 contains 302,175 samples.
For each of the three datasets, we randomly select 80% for training, 10% as the validation set and the rest 10% for evaluation

Add Lipophilicity Dataset to HuggingFace

Found here: https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#lipo-datasets

• Add selfies column

Add ClinTox Dataset to HuggingFace

Found here: https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#clintox-datasets

Only need the clinical toxicity (CT_TOX) task from ClinTox

• Add selfies column

Add Clearance Dataset to Huggingface

Found here: https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#clearance-datasets

• Add selfies column

Add BBBP Dataset to HuggingFace

Found here: https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#bbbp-datasets

• Add selfies column