Suggestion: Package manager about pysrurgs

[pySRURGS]

Since pySRURGS saves files within its own directory and uses relative paths a lot, I have some hesitation about making it a package, as that may need a good deal of re-writing. I imagine that this would entail loading the package then specifying a working directory in which pySRURGS can save files. I'll leave this issue open and may work on it later.

[praksovar]

Hello,
Thank you very much for your software. I appreciate the speed and multiprocessing! But I found some problems during the computing on my dataset. I ran: "python pySRURGS.py -max_permitted_trees 1000 -path_to_db ./db/my_database_autocomm.db ./csv/all_sorted.csv 2000"
After approx. 1000 iterations the error of "database locked" occurs (the copy below). I also tested the same task on a cluster with 36 CPUs and the same error occured. I also used a ramdisk on Linux Mint or '":memory:" as a path to database, but nothing changed, the same error occured. I think that the error is caused by sqlite3. Please, could you give me some advises, or would you be so kind and edit little bit your software e.g. changing the type of database (PostgreSQL, ...)?
Thank you in advance.
Renata

MY ERROR:

Exception in thread Thread-671:
Traceback (most recent call last):
File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run
conn.execute('PRAGMA journal_mode = %s' % self.journal_mode)
sqlite3.OperationalError: database is locked

Exception in thread Thread-666:
Traceback (most recent call last):
File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run
conn.execute('PRAGMA journal_mode = %s' % self.journal_mode)
sqlite3.OperationalError: database is locked

Exception in thread Thread-619:
Traceback (most recent call last):
File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run
conn.execute('PRAGMA journal_mode = %s' % self.journal_mode)
sqlite3.OperationalError: database is locked

Exception in thread Thread-658:
Traceback (most recent call last):
File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run
conn.execute('PRAGMA journal_mode = %s' % self.journal_mode)
sqlite3.OperationalError: database is locked

[pySRURGS]

This is not a small fix. Sqlite does not support this kind of concurrency. I am doing other projects at the moment. I would suggest you consider the pyGOURGS project, which does not save all results to memory. You can try that project using the symbolic regression script but I am not confident its symbolic regression script is thoroughly tested for multiprocessing. Sohrab Towfighi (sent from cellphone)

…

On Sun, Jan 24, 2021, 11:43 AM praksovar, ***@***.***> wrote: Hello, Thank you very much for your software. I appreciate the speed and multiprocessing! But I found some problems during the computing on my dataset. I ran: "python pySRURGS.py -max_permitted_trees 1000 -path_to_db ./db/my_database_autocomm.db ./csv/all_sorted.csv 2000" After approx. 1000 iterations the error of "database locked" occurs (the copy below). I also tested the same task on a cluster with 36 CPUs and the same error occured. I also used a ramdisk on Linux Mint or '":memory:" as a path to database, but nothing changed, the same error occured. I think that the error is caused by sqlite3. Please, could you give me some advises, or would you be so kind and edit little bit your software e.g. changing the type of database (PostgreSQL, ...)? Thank you in advance. Renata MY ERROR: Exception in thread Thread-671: Traceback (most recent call last): File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run conn.execute('PRAGMA journal_mode = %s' % self.journal_mode) sqlite3.OperationalError: database is locked Exception in thread Thread-666: Traceback (most recent call last): File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run conn.execute('PRAGMA journal_mode = %s' % self.journal_mode) sqlite3.OperationalError: database is locked Exception in thread Thread-619: Traceback (most recent call last): File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run conn.execute('PRAGMA journal_mode = %s' % self.journal_mode) sqlite3.OperationalError: database is locked Exception in thread Thread-658: Traceback (most recent call last): File "/home/pavel/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/home/pavel/.local/lib/python3.7/site-packages/sqlitedict.py", line 396, in run conn.execute('PRAGMA journal_mode = %s' % self.journal_mode) sqlite3.OperationalError: database is locked — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMZFMBNAUTRBDFNRZIEG6HLS3RZ53ANCNFSM4ISCJ2EQ> .

[pySRURGS]

The issue likely has to do with the filesystem used by high performance computing clusters. The Networked File System is not supported by Sqlite3.

[pySRURGS]

Closing since I do not plan on making this a package managed software.