Problem running Higashi on Ramani et al. about higashi

[ruochiz]

Hi, would be glad to help. Could you provide more info on the code and how you run Higashi?

[sdontsay]

Of course. I run Higashi and just followed your tutorials on Ramani et al. dataset with the script as follows,

import os, sys
import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
from higashi.Higashi_wrapper import *
config = "/home/.../ram_data/config_ramani_100k.JSON"

# Initialize the Higashi instance
higashi_model = Higashi(config)

# Data processing (only needs to be run for once)
higashi_model.process_data()
higashi_model.prep_model()

# Stage 1 training
# higashi_model.train_for_embeddings()

# Stage 2 training and imputation without neighbor information
higashi_model.train_for_imputation_nbr_0()
higashi_model.impute_no_nbr()

# Stage 3 training and imputation with neighbor information
higashi_model.train_for_imputation_with_nbr()
higashi_model.impute_with_nbr()

and I am submitting the job on an HPC through Slurm, nothing special, I also tried it on another HPC server and got the same error message.

The configuration file is as follows in case you wanna have a look,
{
"config_name": "ramani et. al",
"data_dir": "/home/.../ram_data",
"temp_dir": "/home/.../Ramani_100k",
"genome_reference_path": "/home/.../4DN_data/hg19.chrom.sizes.txt",
"cytoband_path": "/home/.../4DN_data/cytoBand.txt",
"chrom_list": ["chr1","chr2","chr3","chr4","chr5",
"chr6","chr7","chr8","chr9","chr10",
"chr11","chr12","chr13","chr14","chr15",
"chr16","chr17","chr18","chr19","chr20",
"chr21","chr22","chrX"],
"resolution": 100000,
"resolution_cell": 100000,
"minimum_distance": 200000,
"maximum_distance": -1,
"local_transfer_range": 1,
"dimensions": 64,
"impute_list":["chr1","chr2","chr3","chr4","chr5",
"chr6","chr7","chr8","chr9","chr10",
"chr11","chr12","chr13","chr14","chr15",
"chr16","chr17","chr18","chr19","chr20",
"chr21","chr22","chrX"],
"minimum_impute_distance": 0,
"maximum_impute_distance": -1,
"neighbor_num": 4,
"plot_start": 0,
"plot_end": -1,
"plot_label": ["cell type"],
"call_tads": false,
"embedding_name": "exp_zinb3",
"cpu_num_torch": -1,
"cpu_num": 1,
"gpu_num": -1,
"no_nbr_epoch": 45,
"with_nbr_epoch": 30,
"embedding_epoch": 60,
"optional_smooth": false,
"optional_quantile": false,
"rank_thres": 1,
"loss_mode": "zinb",
"random_walk": false,
"UMAP_params": {"n_neighbors": 20
}
}

The main codes for running Higashi were recently downloaded from your Github, I followed the instructions on installing it, and I did not make any changes.

Moreover, below are some error messages printed out right before the part I attached previously for you as a supplement.

creating matrices tasks: 100%|██████████| 23/23 [01:23<00:00, 8.25s/it]
creating matrices tasks: 100%|██████████| 23/23 [01:23<00:00, 3.62s/it]

0%| | 0/23 [00:00<?, ?it/s]
4%|▍ | 1/23 [00:01<00:36, 1.66s/it]
9%|▊ | 2/23 [00:03<00:31, 1.50s/it]
13%|█▎ | 3/23 [00:04<00:26, 1.33s/it]
17%|█▋ | 4/23 [00:04<00:21, 1.13s/it]
22%|██▏ | 5/23 [00:06<00:20, 1.13s/it]
26%|██▌ | 6/23 [00:06<00:17, 1.02s/it]
30%|███ | 7/23 [00:07<00:15, 1.05it/s]
35%|███▍ | 8/23 [00:08<00:13, 1.13it/s]
39%|███▉ | 9/23 [00:09<00:10, 1.29it/s]
43%|████▎ | 10/23 [00:09<00:09, 1.39it/s]
48%|████▊ | 11/23 [00:10<00:08, 1.45it/s]
52%|█████▏ | 12/23 [00:10<00:07, 1.50it/s]
57%|█████▋ | 13/23 [00:11<00:05, 1.71it/s]
61%|██████ | 14/23 [00:11<00:04, 1.92it/s]
65%|██████▌ | 15/23 [00:12<00:03, 2.06it/s]
70%|██████▉ | 16/23 [00:12<00:02, 2.47it/s]
74%|███████▍ | 17/23 [00:12<00:02, 2.87it/s]
78%|███████▊ | 18/23 [00:12<00:01, 3.29it/s]
83%|████████▎ | 19/23 [00:12<00:00, 4.08it/s]
87%|████████▋ | 20/23 [00:12<00:00, 4.37it/s]
96%|█████████▌| 22/23 [00:13<00:00, 6.21it/s]
100%|██████████| 23/23 [00:13<00:00, 3.48it/s]
100%|██████████| 23/23 [00:13<00:00, 1.66it/s]

0%| | 0/300 [00:00<?, ?it/s]
1.012: 0%| | 1/300 [00:01<08:04, 1.62s/it]
0.583: 37%|███▋ | 111/300 [00:01<00:02, 89.61it/s]
0.551: 45%|████▌ | 136/300 [00:01<00:02, 77.92it/s]

0%| | 0/23 [00:00<?, ?it/s]
100%|██████████| 23/23 [00:00<00:00, 564146.15it/s]

0%| | 0/23 [00:00<?, ?it/s]
100%|██████████| 23/23 [00:00<00:00, 551251.38it/s]
Traceback (most recent call last):
File "/home/bin.zhao/Higashi/higashi.py", line 19, in

Please let me know for any further questions.

Thanks

[ruochiz]

Did you ever run the higashi_model.train_for_embeddings() part? It needs to be ruined at least once on a dataset to have the _step1 model that's required for the training for imputation.

[sdontsay]

I cannot believe I forgot to uncomment that code ...

Sorry, Ruochi, I should double-check the script. I commented that step as I did not need it at the time when I run Higashi again on the same data set with the same parameters, and I totally forgot it now. It is working now, thanks a lot!

Best

[ruochiz]

Cool!