"OperationalError: attempt to write a readonly database" and others about trifusion

[MafaldaSFerreira]

Hi again,

In the meantime other TriSeq commands that I was running on smaller alignments (50 kb alignments from across the genome) also started to fail with a different (but possibly related?) error. I'm going to post this error here as well in case it is somehow related:

Executing TriSeq module at 03/07/2020 11:03:34
Parsing 1 alignments
Traceback (most recent call last):
  File "/home/mf239/.local/lib/python2.7/site-packages/trifusion/process/base.py", line 142, in __call__
    self.func(*args)
  File "/home/mf239/.local/lib/python2.7/site-packages/trifusion/TriSeq.py", line 146, in main_parser
    pbar=pbar)
  File "/home/mf2396/.local/lib/python2.7/site-packages/trifusion/process/sequence.py", line 2995, in __init__
    index=("main_idx", "aln_idx"))
  File "/home/mf239/.local/lib/python2.7/site-packages/trifusion/process/sequence.py", line 3292, in _create_table
    "{})".format(table_name, cols))
OperationalError: database is locked
Traceback (most recent call last):
  File "/home/mf239/.local/bin/TriSeq", line 8, in <module>
    sys.exit(main())
  File "/home/mf239/.local/lib/python2.7/site-packages/trifusion/TriSeq.py", line 510, in main
    main_parser(arguments, arguments.infile)
  File "/home/mf239.local/lib/python2.7/site-packages/trifusion/process/base.py", line 166, in __call__
    shutil.rmtree(self.temp_dir)
  File "/home/mf239/anaconda3/envs/GG/lib/python2.7/shutil.py", line 279, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/mf239/anaconda3/envs/GG/lib/python2.7/shutil.py", line 277, in rmtree
    os.rmdir(path)
OSError: [Errno 2] No such file or directory: '.trifusion-temp' 

Again, thank you!
Mafalda

[fernandoblalves]

Dear Mafalda,

Let's try to cover some basics possible errors first.
TriFusion uses ~/.trifusion to store some files. Do you have enough space in your home folder to accommodate temporary files and databases?
Also, it is possible that SQLite uses /tmp in its internal operations. Is there enough space in there as well?

Best

[MafaldaSFerreira]

Thank you for responding.

I cannot find a ~/.trifusion/ directory in ~/. I installed TriFusion with pip2 since the default python on the server is python3. So, the directory I can find with the same name is under ~/.local/lib/python2.7/site-packages/trifusion. These are its contents:

ls .local/lib/python2.7/site-packages/trifusion
app.py   benchmarks_mem.py   benchmarks.pyc  __init__.pyc          orthomcl_pipeline.pyc  trifusion.kv   TriOrtho.py   TriSeq.pyc
app.pyc  benchmarks_mem.pyc  data            ortho                 process                TriFusion.py   TriOrtho.pyc  TriStats.py
base     benchmarks.py       __init__.py     orthomcl_pipeline.py  progressbar            TriFusion.pyc  TriSeq.py     TriStats.pyc

Would this be the correct directory? If so, there should be enough space (180T) to right here, since this is within my home folder.

As for the /tmp folder, I don't have permission to check how much available space there is in it, but this could be the problem. We recurrently run out of space in the /tmp folder, since it's shared by everyone. To solve this issue before, I've specified a /tmp folder in my home or scratch every time the program allows to do so (with a flag, for example). I couldn't find that option in the list of flags when I call TriSeq, but is there a possibility to specify the path to a temporary folder? I run my jobs on a server managed by slurm, and it seems that I could specify a $TMPDIR variable to force programs to write somewhere different than /tmp (example). Do you think this could work for TriSeq?

Thanks,
Mafalda

[ODiogoSilva]

Hi Mafalda,

I'll try to address all the issues you raised. The first one, with the error OperationalError: attempt to write a readonly database, this is because TriFusion programs were not designed to be run in parallele. TriFusion relies on an sqlite database internally and expects to be the only process access that tdatabase (hence the readonly database error). Unfortunatelly, there is no workaround at the moment, which means that multiple runs of TriSeq/TriStats must be sequentially.

Regarding the OperationalError: database or disk is full error, that's very odd indeed. As @fernandoblalves mentioned, it may have to do with the location of the sqlite database (and possibly some internal usage of TMP by sqlite). TriFusion will store the database locally in ${HOME}/.trifusion. I'm not sure whas is set as $HOME on your cluster, but if that variable is set, then it should be there (unless something very wrong happened in the previous execution). There is also the possibility that sqlite is using temporary data internally in a location where it runs out of space. According to sqlite documentation, there are these possible locations: https://sqlite.org/tempfiles.html#tempdir (under Temporary File Storage Locations). Are you able to check if you have access to those locations and if there is enough space in them?