Failing to run SigProfilerExtractor about sigprofilerextractor

[Aikaterini-Chatzipli]

Thank you very much, Mark. The Fonts issue is resolved but I am now getting a different error. Please read below. The run_subs_signatures.py I run contains your code and I attach it below too. Thanks.

---

from SigProfilerExtractor import sigpro as sig

subsVcfFolder='/hps/filt_vcfs/subs/'

toti = 50

output_folder='/hps/subsResults/'

sig.sigProfilerExtractor("vcf",
output_folder,
subsVcfFolder,
reference_genome="GRCh38",
minimum_signatures=1,
maximum_signatures=8,
nmf_replicates=toti)

Traceback (most recent call last):
File "run_subs_signatures.py", line 1, in
from SigProfilerExtractor import sigpro as sig
File "/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/sigpro.py", line 33, in
from SigProfilerExtractor import subroutines as sub
File "/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/subroutines.py", line 26, in
import PyPDF2
ModuleNotFoundError: No module named 'PyPDF2'

[mishugeb]

Please update the package from GitHub

[Aikaterini-Chatzipli]

Was there a bug and have you fixed it now?

Thanks

[mishugeb]

There was initially a bug. But it should work now.

[Aikaterini-Chatzipli]

I updated the software but I am still getting the same error as before.

[mishugeb]

Hi Mark, I believe the issue the user is having came from the decomposition plot. Given that I don't know any other user having the issue, do you have any idea on the cause of the issue for this particular user? Thanks, Mishu

…

________________________________ From: Aikaterini-Chatzipli <[email protected]> Sent: Tuesday, June 23, 2020 3:18 PM To: AlexandrovLab/SigProfilerExtractor <[email protected]> Cc: Islam, S <[email protected]>; Comment <[email protected]> Subject: Re: [AlexandrovLab/SigProfilerExtractor] Failing to run SigProfilerExtractor (#29) I updated the software but I am still getting the same error as before. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#29 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFOQ2HLGZM7U4F3TQJGH3T3RYES3DANCNFSM4OC3OI4Q>.

[mdbarnesUCSD]

Hello Katerina,

It looks like ReportLab is unable to find the 'Arial Bold.ttf' font file. I would like to verify that the 'Arial Bold.ttf' font file exists and is accessible on your system.

Could you please check that the version you have downloaded has a 'Fonts' directory and that it contains 'Arial Bold.ttf'? I believe you can check it exists here:
ls /hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/src/Fonts

If it does have a 'Fonts' directory, could you please check that the 'Arial Bold.ttf' file inside of it works? You can test it by running this code in a python instance (note: reportlab.rl_config.TTFSearchPath is where ReportLab is going to search for the font file):

import reportlab
from reportlab.pdfbase.ttfonts import TTFont
from reportlab.pdfbase import pdfmetrics
reportlab.rl_config.TTFSearchPath = ["/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/src/Fonts"]
pdfmetrics.registerFont(TTFont('Arial-Bold', 'Arial Bold.ttf'))

Please let me know if you come across any issues.

Thanks,
Mark

[Aikaterini-Chatzipli]

Hi Mark,

Thanks for you message. It seems I don't have the Fonts directory. How can I get it?

Thanks,

Katerina

[mdbarnesUCSD]

The Fonts directory is included in the current version of the GitHub repository. Could you please update your environment's version of SigProfilerExtractor with the current version of SigProfilerExtractor on GitHub?

[Aikaterini-Chatzipli]

I did update the package from GitHub using the following line:
"git clone https://github.com/AlexandrovLab/SigProfilerExtractor.git"
Then, I activated the environment and run SigProfilerExtractor but I got the same error about the Fonts. Do you think the environment is not somehow updated? If so, what else should I do?

Thank you,
Katerina

[mdbarnesUCSD]

I believe that you are running SigProfilerExtractor from the virtual environment named "SigProfilerExtractor":
/hps/anaconda3/envs/SigProfilerExtractor/

The virtual environment has the GitHub package SigProfilerExtractor installed at:
/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor

In order to update the version of SigProfilerExtractor that your virtual environment is using, you will need to update the SigProfilerExtractor library at:
/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor

This is a way to update the SigProfilerExtractor library in your virtual environment

After running:
git clone https://github.com/AlexandrovLab/SigProfilerExtractor.git

You will want to copy the file over to your virtual environment's site-packages:
cp -r SigProfilerExtract/SigProfilerExtractor /hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/

Please let me know if this works for you.

Thanks,
Mark

[mishugeb]

I am curious why your environment is not automatically installing the requirements when you install the SigProfilerExtractor tool. Anyway, just write "$pip install PyPDF2" in your terminal. That should solve the problem.

[Aikaterini-Chatzipli]

Yes, this is solved, thank you. I have got two more though, please see below. Is it supposed to find log files which do not exist?

os.remove(log_file)

FileNotFoundError: [Errno 2] No such file or directory: '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-06-25.out'

os.remove(error_file)

FileNotFoundError: [Errno 2] No such file or directory: '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-06-25.err'

[mishugeb]

Hi Erik, Could you please check if the issue the user is talking about is coming from MatrixGenerator. Thanks, Mishu

…

________________________________ From: Aikaterini-Chatzipli <[email protected]> Sent: Thursday, June 25, 2020 11:41 AM To: AlexandrovLab/SigProfilerExtractor <[email protected]> Cc: Islam, S <[email protected]>; Comment <[email protected]> Subject: Re: [AlexandrovLab/SigProfilerExtractor] Failing to run SigProfilerExtractor (#29) Yes, this is solved, thank you. I have got two more though, please see below. Is it supposed to find log files which do not exist? os.remove(log_file) FileNotFoundError: [Errno 2] No such file or directory: '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-06-25.out' os.remove(error_file) FileNotFoundError: [Errno 2] No such file or directory: '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-06-25.err' — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#29 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFOQ2HJJPLWSOCMW3W6OUZDRYOK57ANCNFSM4OC3OI4Q>.

[ebergstr]

Hi,

Can I please ask that you update SigProfilerMatrixGenerator. This appears to be the result of a bug that was fixed some time ago. If you are still experiencing the problem please let me know and please include the version of SigProfilerMatrixGenerator that is installed on your system (you can type the command: pip show SigProfilerMatrixGenerator)!

Best,
Erik

[Aikaterini-Chatzipli]

Thanks Erik! I updated SigProfilerMatrixGenerator and the version I am using is 1.1.16. It is very strange that the code still complains about the log files which do exist. Please read messages below. What would you advise me to do next?

Thank you,

Katerina

File "/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/sigpro.py", line 543, in sigProfilerExtractor
data = datadump.SigProfilerMatrixGeneratorFunc(project_name, refgen, project, exome=exome, bed_file=None, chrom_based=False, plot=False, gs=False)
File "/hps/anaconda3/envs/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerMatrixGenerator/scripts/SigProfilerMatrixGeneratorFunc.py", line 303, in SigProfilerMatrixGeneratorFunc
os.remove(log_file)
FileNotFoundError: [Errno 2] No such file or directory: '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-06-28.out'

[ebergstr]

Can you please provide all of your commands and/or script that you are using to run the tools along with the directory from which you are running the code (type pwd)? We have not experienced this with any other users, so I imagine there is a simple solution relating to project directory structure or from where you are running the code.

Best,
Erik

[Aikaterini-Chatzipli]

Here is how I updated SigProfilerMatrixGenerator

$ pip install SigProfilerMatrixGenerator --upgrade

Requirement already up-to-date: SigProfilerMatrixGenerator in /hps/anaconda3/lib/python3.7/site-packages (1.1.16)
Requirement already satisfied, skipping upgrade: matplotlib>=2.2.2 in /hps/anaconda3/lib/python3.7/site-packages (from SigProfilerMatrixGenerator) (3.1.3)
Requirement already satisfied, skipping upgrade: pandas>=0.23.4 in /hps/anaconda3/lib/python3.7/site-packages (from SigProfilerMatrixGenerator) (1.0.1)
Requirement already satisfied, skipping upgrade: numpy>=1.14.3 in /hps/anaconda3/lib/python3.7/site-packages (from SigProfilerMatrixGenerator) (1.18.1)
Requirement already satisfied, skipping upgrade: sigProfilerPlotting>=1.0.1 in /hps/anaconda3/lib/python3.7/site-packages (from SigProfilerMatrixGenerator) (1.1.6)
Requirement already satisfied, skipping upgrade: statsmodels>=0.9.0 in /hps/anaconda3/lib/python3.7/site-packages (from SigProfilerMatrixGenerator) (0.11.0)
Requirement already satisfied, skipping upgrade: scipy>=1.1.0 in /hps/anaconda3/lib/python3.7/site-packages (from SigProfilerMatrixGenerator) (1.4.1)
Requirement already satisfied, skipping upgrade: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /hps/anaconda3/lib/python3.7/site-packages (from matplotlib>=2.2.2->SigProfilerMatrixGenerator) (2.4.6)
Requirement already satisfied, skipping upgrade: cycler>=0.10 in /hps/anaconda3/lib/python3.7/site-packages (from matplotlib>=2.2.2->SigProfilerMatrixGenerator) (0.10.0)
Requirement already satisfied, skipping upgrade: kiwisolver>=1.0.1 in /hps/anaconda3/lib/python3.7/site-packages (from matplotlib>=2.2.2->SigProfilerMatrixGenerator) (1.1.0)
Requirement already satisfied, skipping upgrade: python-dateutil>=2.1 in /hps/anaconda3/lib/python3.7/site-packages (from matplotlib>=2.2.2->SigProfilerMatrixGenerator) (2.8.1)
Requirement already satisfied, skipping upgrade: pytz>=2017.2 in /hps/anaconda3/lib/python3.7/site-packages (from pandas>=0.23.4->SigProfilerMatrixGenerator) (2019.3)
Requirement already satisfied, skipping upgrade: patsy>=0.5 in /hps/anaconda3/lib/python3.7/site-packages (from statsmodels>=0.9.0->SigProfilerMatrixGenerator) (0.5.1)
Requirement already satisfied, skipping upgrade: six in /hps/anaconda3/lib/python3.7/site-packages (from cycler>=0.10->matplotlib>=2.2.2->SigProfilerMatrixGenerator) (1.14.0)
Requirement already satisfied, skipping upgrade: setuptools in /hps/anaconda3/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib>=2.2.2->SigProfilerMatrixGenerator) (45.2.0.post20200210)

My working directory is /hps/code/ from where I call the script named run_subs_signatures.py which contains the lines below:

`from SigProfilerExtractor import sigpro as sig

subsVcfFolder='/hps/filt_vcfs/subs/'

toti = 50

output_folder='/hps/subsResults/'

sig.sigProfilerExtractor("vcf",
output_folder,
subsVcfFolder,
reference_genome="GRCh38",
minimum_signatures=1,
maximum_signatures=8,
nmf_replicates=toti)`

---

To run run_subs_signatures.py I do the following,

$ pwd
/hps/code/
$ conda activate SigProfilerExtractor
$ python run_subs_signatures.py

[Aikaterini-Chatzipli]

Hi Eric,

Can you see anything obviously wrong based on the information provided in my previous comment?

Thank you,

Katerina

[ebergstr]

I am unsure as to what is causing this issue, as we cannot recreate the error nor has it been reported elsewhere. Can I ask you to try running the matrixGenerator separately? You can do this quite easily from an interactive python session. If this works, you can then run the extractor using the final generated matrices, rather than starting from vcf files each time.

$python3

from SigProfilerMatrixGenerator.scripts import SigProfilerMatrixGeneratorFunc as matGen
matrices = matGen.SigProfilerMatrixGeneratorFunc("hps", "GRCh38", "/hps/filt_vcfs/subs/")

If this give you an error still, you can simply comment out line 303, in SigProfilerMatrixGeneratorFunc:
#os.remove(log_file)

Best,
Erik

[Aikaterini-Chatzipli]

Thanks Erik! I got the log file error below after commenting out line 303 from SigProfilerMatrixGeneratorFunc.py. Looking through SigProfilerMatrixGeneratorFunc.py, I see the following:

300 if os.path.exists(error_file):
301 os.remove(error_file)
302 if os.path.exists(log_file):
303 # os.remove(log_file)
304 sys.stderr = open(error_file, 'w')

#########

Traceback (most recent call last):
File "", line 1, in
File "/hps/anaconda3/lib/python3.7/site-packages/SigProfilerMatrixGenerator/scripts/SigProfilerMatrixGeneratorFunc.py", line 304, in SigProfilerMatrixGeneratorFunc
sys.stderr = open(error_file, 'w')
FileNotFoundError: [Errno 2] No such file or directory: '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_/hps/_GRCh382020-07-08.err'

###########

Also I don't understand why the script looks into '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_/hps/_GRCh382020-07-08.err'

It should be looking into something more like '/hps/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-06-28.err'

What do you think?

Thanks,

Katerina

[ebergstr]

You can also comment out "os.remove(error_file)" as that is now giving you the issue. It appears from the error output that you are using "/hps/" as your project name, hence the tool is looking for a file called "SigProfilerMatrixGenerator_/hps/_GRCh382020-07-08.err." You should use "subs" as your project name if you would like the file to find 'SigProfilerMatrixGenerator_subs_GRCh382020-06-28.err.'

Best,
Erik

[Aikaterini-Chatzipli]

Hi Erik,

I am reopening this ticket as I encountered more problems while running SigProfilerExtractor.

I run SigProfilerExtractor from a fresh installation in a new environment. I got the errors below when I commented out lines 302 and 304 from SigProfilerMatrixGeneratorFunc.py as suggested before.

---

(SigProfilerExtractor) [ac630@compute-e-16-233 scripts]$ python run_subs_signatures.py
Traceback (most recent call last):
File "run_subs_signatures.py", line 4, in
from SigProfilerExtractor import sigpro as sig
File "/n//katerina/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/sigpro.py", line 35, in
from SigProfilerMatrixGenerator.scripts import SigProfilerMatrixGeneratorFunc as datadump
File "/n/katerina/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerMatrixGenerator/scripts/SigProfilerMatrixGeneratorFunc.py", line 302
if os.path.exists(log_file):
^
IndentationError: expected an indented block
(SigProfilerExtractor) [ac630@compute-e-16-233 scripts]$ python run_subs_signatures.py
Traceback (most recent call last):
File "run_subs_signatures.py", line 4, in
from SigProfilerExtractor import sigpro as sig
File "/n/katerina/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/sigpro.py", line 35, in
from SigProfilerMatrixGenerator.scripts import SigProfilerMatrixGeneratorFunc as datadump
File "/n/katerina/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerMatrixGenerator/scripts/SigProfilerMatrixGeneratorFunc.py", line 304
sys.stderr = open(error_file, 'w')
^
IndentationError: expected an indented block

---

If I don't comment out lines 302 and 304 from SigProfilerMatrixGeneratorFunc.py, SigProfilerExtractor additionally complains for the following:

Traceback (most recent call last):
File "", line 1, in
File "/n/app/python/3.7.4/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "/n/app/python/3.7.4/lib/python3.7/multiprocessing/spawn.py", line 114, in _main
prepare(preparation_data)
File "/n/app/python/3.7.4/lib/python3.7/multiprocessing/spawn.py", line 225, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/n/app/python/3.7.4/lib/python3.7/multiprocessing/spawn.py", line 277, in _fixup_main_from_path
run_name="mp_main")
File "/n/app/python/3.7.4/lib/python3.7/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/n/app/python/3.7.4/lib/python3.7/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/n/app/python/3.7.4/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/n/katerina/scripts/run_subs_signatures.py", line 20, in
nmf_replicates=toti)
File "/n/katerina/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerExtractor/sigpro.py", line 579, in sigProfilerExtractor
data = datadump.SigProfilerMatrixGeneratorFunc(project_name, refgen, project, exome=exome, bed_file=None, chrom_based=False, plot=False, gs=False)
File "/n/katerina/SigProfilerExtractor/lib/python3.7/site-packages/SigProfilerMatrixGenerator/scripts/SigProfilerMatrixGeneratorFunc.py", line 301, in SigProfilerMatrixGeneratorFunc
os.remove(error_file)
FileNotFoundError: [Errno 2] No such file or directory: '/n/katerina/filt_vcfs/subs/logs/SigProfilerMatrixGenerator_subs_GRCh382020-07-19.err'

---

SigProfilerExtractor creates a logs directory in the input file directory. It generates the log files in there which later on reports as not found. I am not sure why it does that as I can see the files exist. Your input will be much appreciated.

Many thanks,
Katerina

[amootta]

Hi Erik,
Have been having the same problem as Katerina above and I have been trying to troubleshoot it to no avail. I have the newest versions of SigProfile so I'm not entirely sure why these errors are still popping up? please let me know how best to fix it. The error is that it is not recognizing the report (.err) file despite it existing in the exact same path it is looking for.
Thanks
Amatta