I was trying out atlas_analyze. So far so good, I have only one error at the second last step in rule convert_nb.
Here is the error line:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 all
1 convert_nb
2
Select jobs to execute...
[Sat Jan 16 16:07:43 2021]
rule convert_nb:
input: Results/Code.ipynb
output: Results/Summary.html
jobid: 1
[NbConvertApp] Converting notebook Results/Code.ipynb to html
Traceback (most recent call last):
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/reader.py", line 14, in parse_json
nb_dict = json.loads(s, **kwargs)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/json/__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/bin/jupyter-nbconvert", line 11, in <module>
sys.exit(main())
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/jupyter_core/application.py", line 254, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/traitlets/config/application.py", line 845, in launch_instance
app.start()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 350, in start
self.convert_notebooks()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
self.convert_single_notebook(notebook_filename)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
return self.from_file(f, resources=resources, **kw)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/__init__.py", line 143, in read
return reads(buf, as_version, **kwargs)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/__init__.py", line 73, in reads
nb = reader.reads(s, **kwargs)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/reader.py", line 58, in reads
nb_dict = parse_json(s, **kwargs)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/reader.py", line 17, in parse_json
raise NotJSONError(("Notebook does not appear to be JSON: %r" % s)[:77] + "...") from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: ''...
[Sat Jan 16 16:07:49 2021]
Error in rule convert_nb:
jobid: 1
output: Results/Summary.html
shell:
jupyter nbconvert --output Summary --to=html --TemplateExporter.exclude_input=True Results/Code.ipynb
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/.snakemake/log/2021-01-16T160743.880488.snakemake.log
Traceback (most recent call last):
File "./analyze.py", line 21, in <module>
"snakemake "
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/shell.py", line 213, in __new__
raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail; snakemake -d /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena -j 1 -s ./Snakefile' returned non-zero exit status 1.
[Sat Jan 16 16:05:35 2021]
localrule analyze:
input: Results/taxonomy.tsv, Results/mapping_rate.tsv, Results/genome_completeness.tsv, Results/counts/raw_counts_genomes.tsv, Results/counts/median_coverage_genomes.tsv, Results/annotations/KO.tsv, Results/annotations/CAZy.tsv, Results/annotations/KO.tsv, Results/annotations/CAZy.tsv
log: Results/Code.ipynb
jobid: 2
[NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)
Failed validating 'additionalProperties' in code_cell:
On instance['cells'][0]:
{'cell_type': 'code',
'execution_count': None,
'id': 'hourly-liberal',
'metadata': {'tags': ['snakemake-job-properties']},
'outputs': ['...0 outputs...'],
'source': '\n'
'######## snakemake preamble start (automatically inserted, do '
'n...'}
Traceback (most recent call last):
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/bin/jupyter-nbconvert", line 11, in <module>
sys.exit(main())
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/jupyter_core/application.py", line 254, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/traitlets/config/application.py", line 845, in launch_instance
app.start()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 350, in start
self.convert_notebooks()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
self.convert_single_notebook(notebook_filename)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
return self.from_file(f, resources=resources, **kw)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
nb_copy, resources = self._preprocess(nb_copy, resources)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
nbc, resc = preprocessor(nbc, resc)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
return self.preprocess(nb, resources)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
self.execute()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
return just_run(coro(*args, **kwargs))
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
return loop.run_until_complete(coro)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
return future.result()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/client.py", line 541, in async_execute
cell, index, execution_count=self.code_cells_executed + 1
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 123, in async_execute_cell
cell, resources = self.preprocess_cell(cell, self.resources, cell_index)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 146, in preprocess_cell
cell = run_sync(NotebookClient.async_execute_cell)(self, cell, index, store_history=self.store_history)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
return just_run(coro(*args, **kwargs))
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
return loop.run_until_complete(coro)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/asyncio/futures.py", line 181, in result
raise self._exception
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/asyncio/tasks.py", line 249, in __step
result = coro.send(None)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/client.py", line 832, in async_execute_cell
self._check_raise_for_error(cell, exec_reply)
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/client.py", line 740, in _check_raise_for_error
raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
#CAZy
CAZy_annotations_genome= pd.read_table('Results/annotations/CAZy.tsv',index_col=0)
CAZy_presence= (CAZy_annotations_genome>0).astype(int)
CAZy_presence.head()
function_relab = relab @ CAZy_presence
sns.clustermap(function_relab)
function_relab.head()
------------------
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-1-87471da32f9f> in <module>
7 function_relab = relab @ CAZy_presence
8
----> 9 sns.clustermap(function_relab)
10
11 function_relab.head()
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/_decorators.py in inner_f(*args, **kwargs)
44 )
45 kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 46 return f(**kwargs)
47 return inner_f
48
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in clustermap(data, pivot_kws, method, metric, z_score, standard_scale, figsize, cbar_kws, row_cluster, col_cluster, row_linkage, col_linkage, row_colors, col_colors, mask, dendrogram_ratio, colors_ratio, cbar_pos, tree_kws, **kwargs)
1410 row_cluster=row_cluster, col_cluster=col_cluster,
1411 row_linkage=row_linkage, col_linkage=col_linkage,
-> 1412 tree_kws=tree_kws, **kwargs)
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in plot(self, metric, method, colorbar_kws, row_cluster, col_cluster, row_linkage, col_linkage, tree_kws, **kws)
1221 self.plot_dendrograms(row_cluster, col_cluster, metric, method,
1222 row_linkage=row_linkage, col_linkage=col_linkage,
-> 1223 tree_kws=tree_kws)
1224 try:
1225 xind = self.dendrogram_col.reordered_ind
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in plot_dendrograms(self, row_cluster, col_cluster, metric, method, row_linkage, col_linkage, tree_kws)
1067 self.data2d, metric=metric, method=method, label=False, axis=0,
1068 ax=self.ax_row_dendrogram, rotate=True, linkage=row_linkage,
-> 1069 tree_kws=tree_kws
1070 )
1071 else:
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/_decorators.py in inner_f(*args, **kwargs)
44 )
45 kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 46 return f(**kwargs)
47 return inner_f
48
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in dendrogram(data, linkage, axis, label, metric, method, rotate, tree_kws, ax)
774 plotter = _DendrogramPlotter(data, linkage=linkage, axis=axis,
775 metric=metric, method=method,
--> 776 label=label, rotate=rotate)
777 if ax is None:
778 ax = plt.gca()
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in __init__(self, data, linkage, metric, method, axis, label, rotate)
582
583 if linkage is None:
--> 584 self.linkage = self.calculated_linkage
585 else:
586 self.linkage = linkage
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in calculated_linkage(self)
649 warnings.warn(msg)
650
--> 651 return self._calculate_linkage_scipy()
652
653 def calculate_dendrogram(self):
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in _calculate_linkage_scipy(self)
618 def _calculate_linkage_scipy(self):
619 linkage = hierarchy.linkage(self.array, method=self.method,
--> 620 metric=self.metric)
621 return linkage
622
/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/scipy/cluster/hierarchy.py in linkage(y, method, metric, optimal_ordering)
1063
1064 if not np.all(np.isfinite(y)):
-> 1065 raise ValueError("The condensed distance matrix must contain only "
1066 "finite values.")
1067
ValueError: The condensed distance matrix must contain only finite values.
ValueError: The condensed distance matrix must contain only finite values.
[Sat Jan 16 16:06:24 2021]
Error in rule analyze:
jobid: 2
log: Results/Code.ipynb (check log file(s) for error message)
RuleException:
CalledProcessError in line 82 of /ddn1/vol1/site_scratch/leuven/314/vsc31426/atlas_analyse/atlas_analyze/Snakefile:
Command 'set -euo pipefail; jupyter-nbconvert --log-level ERROR --execute --output /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/Results/Code.ipynb --to notebook --ExecutePreprocessor.timeout=-1 /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/.snakemake/scripts/tmpbmbi86fl.Analyis_genome_abundances.ipynb' returned non-zero exit status 1.
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2338, in run_wrapper
File "/ddn1/vol1/site_scratch/leuven/314/vsc31426/atlas_analyse/atlas_analyze/Snakefile", line 82, in __rule_analyze
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 566, in _callback
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/concurrent/futures/thread.py", line 57, in run
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 552, in cached_or_run
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2350, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/.snakemake/log/2021-01-16T160453.469631.snakemake.log
Traceback (most recent call last):
File "./analyze.py", line 21, in <module>
"snakemake "
File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/shell.py", line 213, in __new__
raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail; snakemake -d /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena -j 1 -s ./Snakefile' returned non-zero exit status 1.