Dir sir,
I am trying to split the sample index added to cells using the order "multi_tag", and the used tags were tag 8,9,10,11. However, after the command, the output was incomplete, despite the tag.fasta and tag.mapfile seems no problems. The log file of this command is as fellow:
(celescope) [zxia@localhost shell]$ bash sample_index.sh
2022-05-25 17:06:57,277 - celescope.tools.sample.sample - INFO - start...
2022-05-25 17:06:57,289 - celescope.tools.sample.run - INFO - start...
2022-05-25 17:06:57,289 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:57,289 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000217
2022-05-25 17:06:57,306 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:57,308 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.002502
2022-05-25 17:06:57,310 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:57,321 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.010549
2022-05-25 17:06:57,321 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:57,321 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000017
2022-05-25 17:06:57,322 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:57,325 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.002395
2022-05-25 17:06:57,326 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:57,366 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.040109
2022-05-25 17:06:57,367 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:57,367 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000020
2022-05-25 17:06:57,369 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:57,380 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.011256
2022-05-25 17:06:57,382 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:57,422 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.040039
2022-05-25 17:06:57,423 - celescope.tools.barcode.check_chemistry - INFO - start...
/home/data/mybgi/Ischemic_2/Gex_lib/SI_lib_R1.fq.gz
2022-05-25 17:06:57,423 - celescope.tools.barcode.get_chemistry - INFO - start...
2022-05-25 17:06:57,487 - celescope.tools.barcode.get_chemistry - INFO - [('scopeV2.2.1', 8050), ('scopeV2.1.1', 1967)]
2022-05-25 17:06:57,487 - celescope.tools.barcode.get_chemistry - INFO - chemistry: scopeV2.2.1
2022-05-25 17:06:57,487 - celescope.tools.barcode.get_chemistry - INFO - done. time used: 0:00:00.064431
2022-05-25 17:06:57,487 - celescope.tools.barcode.check_chemistry - INFO - done. time used: 0:00:00.064540
2022-05-25 17:06:57,491 - celescope.tools.sample.run - INFO - done. time used: 0:00:00.201875
2022-05-25 17:06:57,491 - celescope.tools.step._clean_up - INFO - start...
2022-05-25 17:06:57,491 - celescope.tools.step._render_html - INFO - start...
2022-05-25 17:06:57,775 - celescope.tools.step._render_html - INFO - done. time used: 0:00:00.283790
2022-05-25 17:06:57,775 - celescope.tools.step._clean_up - INFO - done. time used: 0:00:00.284662
2022-05-25 17:06:57,776 - celescope.tools.sample.sample - INFO - done. time used: 0:00:00.497986
2022-05-25 17:06:59,474 - celescope.tools.barcode.barcode - INFO - start...
2022-05-25 17:06:59,486 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:59,486 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000212
2022-05-25 17:06:59,495 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,497 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.002480
2022-05-25 17:06:59,499 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,509 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.010096
2022-05-25 17:06:59,509 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:59,509 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000017
2022-05-25 17:06:59,510 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,513 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.002442
2022-05-25 17:06:59,514 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,555 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.040810
2022-05-25 17:06:59,556 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:59,556 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000020
2022-05-25 17:06:59,557 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,569 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.011883
2022-05-25 17:06:59,571 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,610 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.039437
2022-05-25 17:06:59,611 - celescope.tools.barcode.check_chemistry - INFO - start...
/home/data/mybgi/Ischemic_2/Gex_lib/SI_lib_R1.fq.gz
2022-05-25 17:06:59,611 - celescope.tools.barcode.get_chemistry - INFO - start...
2022-05-25 17:06:59,658 - celescope.tools.barcode.get_chemistry - INFO - [('scopeV2.2.1', 8050), ('scopeV2.1.1', 1967)]
2022-05-25 17:06:59,658 - celescope.tools.barcode.get_chemistry - INFO - chemistry: scopeV2.2.1
2022-05-25 17:06:59,658 - celescope.tools.barcode.get_chemistry - INFO - done. time used: 0:00:00.047228
2022-05-25 17:06:59,658 - celescope.tools.barcode.check_chemistry - INFO - done. time used: 0:00:00.047331
2022-05-25 17:06:59,662 - celescope.tools.barcode.run - INFO - start...
2022-05-25 17:06:59,662 - celescope.tools.barcode.run - INFO - lowQual score: 0
2022-05-25 17:06:59,662 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 17:06:59,662 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000021
2022-05-25 17:06:59,664 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,666 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.002343
2022-05-25 17:06:59,668 - celescope.tools.barcode.get_mismatch_dict - INFO - start...
2022-05-25 17:06:59,708 - celescope.tools.barcode.get_mismatch_dict - INFO - done. time used: 0:00:00.039685
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - /home/data/mybgi/Ischemic_2/Gex_lib/SI_lib_R1.fq.gz finished.
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - processed reads: 1,191,266,263. valid reads: 0.
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - no polyT reads number : 1191266263
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - low qual reads number: 0
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - no_linker: 0
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - no_barcode: 0
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - corrected linker: 0
2022-05-25 18:48:26,357 - celescope.tools.barcode.run - INFO - corrected barcode: 0
2022-05-25 18:48:26,357 - celescope.tools.step._clean_up - INFO - start...
2022-05-25 18:48:26,358 - celescope.tools.step._render_html - INFO - start...
2022-05-25 18:48:26,667 - celescope.tools.step._render_html - INFO - done. time used: 0:00:00.309175
2022-05-25 18:48:26,668 - celescope.tools.step._clean_up - INFO - done. time used: 0:00:00.310214
Traceback (most recent call last):
File "/home/zxia/miniconda3/envs/celescope/bin/celescope", line 8, in
sys.exit(main())
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/celescope.py", line 38, in main
args.func(args)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/utils.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/barcode.py", line 563, in barcode
runner.run()
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/utils.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/barcode.py", line 523, in run
'no valid reads found! please check the --chemistry parameter.')
Exception: no valid reads found! please check the --chemistry parameter.
2022-05-25 18:48:28,670 - celescope.tools.cutadapt.cutadapt - INFO - start...
2022-05-25 18:48:28,698 - celescope.tools.cutadapt.run - INFO - start...
2022-05-25 18:48:28,698 - celescope.tools.cutadapt.run - INFO - cutadapt -a polyT=A{18} -a p5=AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC -n 2 -j 4 -m 20 --nextseq-trim=20 --overlap 10 -l 150 -o .//sample_index/02.cutadapt/sample_index_clean_2.fq .//sample_index/01.barcode/sample_index_2.fq
2022-05-25 18:48:28,797 - celescope.tools.step._clean_up - INFO - start...
2022-05-25 18:48:28,797 - celescope.tools.step._render_html - INFO - start...
2022-05-25 18:48:29,077 - celescope.tools.step._render_html - INFO - done. time used: 0:00:00.279351
2022-05-25 18:48:29,078 - celescope.tools.step._clean_up - INFO - done. time used: 0:00:00.281170
Traceback (most recent call last):
File "/home/zxia/miniconda3/envs/celescope/bin/celescope", line 8, in
sys.exit(main())
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/celescope.py", line 38, in main
args.func(args)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/utils.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/cutadapt.py", line 149, in cutadapt
runner.run()
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/utils.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/cutadapt.py", line 143, in run
self.format_and_write_stat(cutadapt_log)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/cutadapt.py", line 75, in format_and_write_stat
total_reads = p_list[0]
IndexError: list index out of range
2022-05-25 18:48:30,703 - celescope.tag.mapping_tag.mapping_tag - INFO - start...
2022-05-25 18:48:30,715 - celescope.tools.barcode.parse_pattern - INFO - start...
2022-05-25 18:48:30,715 - celescope.tools.barcode.parse_pattern - INFO - done. time used: 0:00:00.000204
2022-05-25 18:48:30,716 - celescope.tag.mapping_tag.run - INFO - start...
2022-05-25 18:48:30,730 - celescope.tag.mapping_tag.run - INFO - done. time used: 0:00:00.013526
2022-05-25 18:48:30,730 - celescope.tools.step._clean_up - INFO - start...
2022-05-25 18:48:30,731 - celescope.tools.step._render_html - INFO - start...
2022-05-25 18:48:31,014 - celescope.tools.step._render_html - INFO - done. time used: 0:00:00.283384
2022-05-25 18:48:31,015 - celescope.tools.step._clean_up - INFO - done. time used: 0:00:00.284522
2022-05-25 18:48:31,015 - celescope.tag.mapping_tag.mapping_tag - INFO - done. time used: 0:00:00.311302
Traceback (most recent call last):
File "/home/zxia/miniconda3/envs/celescope/bin/celescope", line 8, in
sys.exit(main())
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/celescope.py", line 38, in main
args.func(args)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tag/count_tag.py", line 54, in count_tag
with Count_tag(args, display_title="Cells") as runner:
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tag/count_tag.py", line 87, in init
self.df_read_count = pd.read_csv(self.read_count_file, sep="\t", index_col=0)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 948, in init
self._make_engine(self.engine)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 2010, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 540, in pandas._libs.parsers.TextReader.cinit
pandas.errors.EmptyDataError: No columns to parse from file
2022-05-25 18:48:34,281 - celescope.tag.analysis_tag.analysis_tag - INFO - start...
Traceback (most recent call last):
File "/home/zxia/miniconda3/envs/celescope/bin/celescope", line 8, in
sys.exit(main())
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/celescope.py", line 38, in main
args.func(args)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tools/utils.py", line 51, in wrapper
result = func(*args, **kwargs)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tag/analysis_tag.py", line 39, in analysis_tag
with Analysis_tag(args, display_title="Analysis") as runner:
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/celescope/tag/analysis_tag.py", line 18, in init
self.df_tsne_tag = pd.read_csv(self.tsne_tag_file, sep='\t')
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 688, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 454, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 948, in init
self._make_engine(self.engine)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 1180, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/home/zxia/miniconda3/envs/celescope/lib/python3.6/site-packages/pandas/io/parsers.py", line 2010, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 382, in pandas._libs.parsers.TextReader.cinit
File "pandas/_libs/parsers.pyx", line 674, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] No such file or directory: './/sample_index/04.count_tag/sample_index_tsne_tag.tsv'
2022-05-25 18:48:36,029 - celescope.tag.split_tag.run - INFO - start...
2022-05-25 18:48:36,029 - celescope.tag.split_tag.run - INFO - done. time used: 0:00:00.000002
2022-05-25 18:48:36,030 - celescope.tools.step._clean_up - INFO - start...
2022-05-25 18:48:36,031 - celescope.tools.step._render_html - INFO - start...
2022-05-25 18:48:36,311 - celescope.tools.step._render_html - INFO - done. time used: 0:00:00.280117
2022-05-25 18:48:36,311 - celescope.tools.step._clean_up - INFO - done. time used: 0:00:00.280994
To help you solve this issue, I also show you some related files:
(celescope) [zxia@localhost sample_tag]$ cat run.sh
source activate celescope
multi_tag
--mapfile /home/data/mybgi/Ischemic_2/celescope_count/sample_tag/tag.mapfile
--mod shell
--barcode_fasta /home/data/mybgi/Ischemic_2/celescope_count/sample_tag/tag.fasta
--fq_pattern L25C15
(celescope) [zxia@localhost sample_tag]$ cat tag.fasta
CLindex_TAG_8
ATACGCAGGGTCCGA
CLindex_TAG_9
AGCGGCATTTGGGAC
CLindex_TAG_10
TCGCCAGCCAAGTCT
CLindex_TAG_11
ACCAATGGCGCATGG
(celescope) [zxia@localhost sample_tag]$ cat tag.mapfile
##tag.mapfile
SI_lib /home/data/mybgi/Ischemic_2/Gex_lib sample_index /home/data/mybgi/Ischemic_2/celescope_count/shell/ischemic_gex
(celescope) [zxia@localhost shell]$ cat sample_index.sh
celescope tag sample --outdir .//sample_index/00.sample --sample sample_index --assay tag --thread 4 --chemistry auto --fq1 /home/data/mybgi/Ischemic_2/Gex_lib/SI_lib_R1.fq.gz
celescope tag barcode --outdir .//sample_index/01.barcode --sample sample_index --assay tag --thread 4 --chemistry auto --lowNum 2 --fq1 /home/data/mybgi/Ischemic_2/Gex_lib/SI_lib_R1.fq.gz --fq2 /home/data/mybgi/Ischemic_2/Gex_lib/SI_lib_R2.fq.gz
celescope tag cutadapt --outdir .//sample_index/02.cutadapt --sample sample_index --assay tag --thread 4 --minimum_length 20 --nextseq_trim 20 --overlap 10 --insert 150 --fq .//sample_index/01.barcode/sample_index_2.fq
celescope tag mapping_tag --outdir .//sample_index/03.mapping_tag --sample sample_index --assay tag --thread 4 --fq_pattern L25C15 --barcode_fasta /home/data/mybgi/Ischemic_2/celescope_count/sample_tag/tag.fasta --fq .//sample_index/02.cutadapt/sample_index_clean_2.fq
celescope tag count_tag --outdir .//sample_index/04.count_tag --sample sample_index --assay tag --thread 4 --UMI_min auto --dim 1 --SNR_min auto --coefficient 0.1 --match_dir /home/data/mybgi/Ischemic_2/celescope_count/shell/ischemic_gex --read_count_file .//sample_index/03.mapping_tag/sample_index_read_count.tsv
celescope tag analysis_tag --outdir .//sample_index/05.analysis_tag --sample sample_index --assay tag --thread 4 --match_dir /home/data/mybgi/Ischemic_2/celescope_count/shell/ischemic_gex --tsne_tag_file .//sample_index/04.count_tag/sample_index_tsne_tag.tsv
celescope tag split_tag --outdir .//sample_index/06.split_tag --sample sample_index --assay tag --thread 4 --match_dir /home/data/mybgi/Ischemic_2/celescope_count/shell/ischemic_gex --umi_tag_file .//sample_index/04.count_tag/sample_index_umi_tag.tsv
I am looking forward for your reply!