jglamine / phamdb

Quickly create and modify Phamerator databases.

PhamDB

A web application for creating and modifying Phamerator databases.

Get started

For installation instructions, see the wiki.

Help and documentation

Help and documentation are available on the wiki.

Support

PhamDB was created by James Lamine as his computer science senior project. His advisors for this project were Serita Nelesen and Randal DeJong. For support questions, please contact Serita Nelesen - snelesen (at) calvin (dot) edu

Features

• Create and manage multiple Phamerator databases
• Upload genbank files to create a database
• Add and remove phages from existing databases
• Analyze your databases in Phamerator
• See live status of running jobs
• Queue multiple jobs
• Search NCBI's conserved domain database

Screenshots

Upload genbank files

See detailed error messages of invalid phages

Analyze your database in Phamerator

Minimum system requirements

Linux

• Docker
• 2GB of RAM (4GB recommended)
• 6GB of hard drive space

Windows and Mac OSX

• Docker toolbox
• 4GB of ram
• 6GB of hard drive space

Recommended

• Dual core processor

For developers

PhamDB is a Python Flask application. It uses Python 2.7, MySQL and RabbitMQ, and calls several Linux command line programs to process gene data.

The easiest way to see how the application is set up is to read the Dockerfile (docker/Dockerfile). It contains information on the various dependencies required. Note that nginx and uwsgi are only needed for production, not development.

Setup

Read the Dockerfile (docker/Dockerfile) to see the basics of how the application is set up.

Among other things, be sure to clone the repository, install the required linux packages, create a python virtual environment, install python packages from requirements.txt, add the project to the virtual environment's PYTHONPATH, run the database creation script, download the conserved domain database using the included makefile (pham/data/conserved-domain-database/Makefile), and verify that all of the tests pass.

Running PhamDB locally

Once you have installed the dependencies and required python packages, you will have to run two commands:

Run the application server with:

python webphamerator/run.py

Run the Celery background job processor with:

celery -A webphamerator.app.celery worker

Testing

From the project's root directory, run the tests with:

nosetests

License

PhamDB is licensed under the GNU GPL, version 3. See license.txt for the full license.

phamdb's Issues

Trouble Opening PhamDB databases in phamerator

Hi there,

Thanks for making PhamDB. It's awesome to be able to make my own databases.
I am trying to open the new databases in the SEA PHAGES VM but when I follow the instructions for adding the server and the database and then closing (whether or not I force database), I get the Database Setup Required, Password prompt from Phamerator. The VM is on line and can access my databases at the server site but not phamerator. I don't have a password on the phamDB databases and this just looks like an error. Am I missing a step?
Thanks!

Heather

Need to install via Docker Desktop

Our biology department got new Windows 10 computers and installed Docker Desktop. They asked me to help load Phamdb because it's not working for them in Docker Desktop. I spent a lot of time trying to get it to work in Docker Desktop as well. Would you be able to provide instructions on how to run Phamdb in Docker Desktop?

"Help" toolbar item which links to the github readme.

failed to run phamdb

hi, i did all the installation steps on Linux,
but i failed to run the image since this step is not clear for me.
can someone help me here?
Regards

How to access database with a password?

I have a Phamerator mysql database created following the BYU Phamerator instructions. How do I add this to phamdb? Thanks!

Jobs stuck in queue

Awesome work on this!

Unfortunately, after going through the create database (made from uploaded validated phage genbanks) step, jobs are waiting in queue forever. Is there a way to see what it is waiting for or a way to manually run the jobs?

PhamDB password issue

Dear All,

I am running PhamDB using docker image. I created a sql database and now when I try to go to http://localhost it asks for password. Could you please tell me what could be the password? I have not set up any password. Thanks so much!

DK

Show confirmation when deleting a database

When deleting a database, show a "Are you sure you want to delete this database?" confirmation modal.

Cannot export phages from old-style databases

For databases imported from SEA phages: http://phamerator.csm.jmu.edu/sea/

Exporting phages from these databases results in a crash.

Related to the closed issues "Jobs stuck in queue #5"

Hello, I understand that this "Jobs stuck in queue #5" issue has been closed. Unfortunately I encountered the same problem two years later. As I am quite new to PhamDB and rabbitmq, could you kindly tell me how I can view the rabbitmq log please? I would like to know if I have the similar "low memory alarm set by rabbitmq".

Sorry for two more simple questions:
1: where can I assess this "mysql> select * from job"?
2: How can I temporarily raise the threshold to 0.7 of total memory :
rabbitmqctl set_vm_memory_high_watermark 0.7

Is it manipulated in the VirtualBox terminal, Docker Toolbox or Kitematic?

Many thanks for your time and kind response.

Phamerator workflow changes

I realize you probably don't maintain this code base much anymore... however, if you do...

I am a graduate student in Graham Hatfull’s lab at the University of Pittsburgh. Because PhamDB makes use of the Phamerator database scheme and parts of our workflow, I wanted to let you know about some changes we’re in the process of making to the Phamerator workflow.

The most important change is that we are ceasing use of kClust and HHsuite in favor of the newer, faster (still Söding-lab produced) MMseqs2. Our testing suggests that this program produces better phams than the iterative kClust approach, in a fraction of the time.

We will also be adding at least one column to the pham table. This column would contain the pham’s conservation score, as assessed by generating a Clustal Omega alignment and dividing the number of perfectly conserved residues by the length of the shortest gene in the pham (this means of calculating prevents draft-status genes, which are frequently called shorter than their manually annotated peers, from artificially lowering the conservation score of the pham). Of course the inclusion of this column means we will also be generating Clustal Omega alignments for each pham moving forward.

I don’t think it’s worth your time to worry about including the Clustal Omega portion of our modified workflow in the PhamDB workflow. However, given that MMseqs2 and kClust work quite differently and result in the production of quite different overall databases, it may be worth it for you or somebody else to update PhamDB to use MMseqs2 instead of kClust. This would likely result in less confusion downstream for Phamerator users trying to build their own databases and wondering why they’re not compatible with https://www.phamerator.org/ or why they can’t get their database clustered as well as we now can.

If you’d like more information about any of these changes or how they may impact you or your users, feel free to reach out.

-Christian Gauthier ([email protected])

Database SQL import times out after 60 seconds

When importing a database as an SQL file, the api request returns 504 after 60 seconds. NGINX is probably timing out the request.

"Create database" fails for unknown reason

Any ideas about where to start troubleshooting?

Creating a database of 32 phages consistently fails at about 27 seconds during step 1 of 2 of calculating phams. A database of 3 phages can be created, no problem.

My OS is Fedora 24, I'm running Docker version 1.10.3, build a612434/1.10.3

Here is what Docker vomits out while running (have run with and without sudo and experience same issue):
docker run --name pham --restart=always -p 8080:80 -v /home/fetz/ssd/docker/phamdb:/dockerdata:Z jglamine/phamdb:latest
[sudo] password for fetz:
2016-09-10 23:45:18,203 CRIT Supervisor running as root (no user in config file)
2016-09-10 23:45:18,203 WARN Included extra file "/etc/supervisor/conf.d/supervisor-app.conf" during parsing
2016-09-10 23:45:18,227 INFO RPC interface 'supervisor' initialized
2016-09-10 23:45:18,227 WARN cElementTree not installed, using slower XML parser for XML-RPC
2016-09-10 23:45:18,227 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-09-10 23:45:18,227 INFO supervisord started with pid 1
2016-09-10 23:45:19,231 INFO spawned: 'celery-app' with pid 8
2016-09-10 23:45:19,232 INFO spawned: 'nginx-app' with pid 9
2016-09-10 23:45:19,235 INFO spawned: 'rabbitmq' with pid 10
2016-09-10 23:45:19,237 INFO spawned: 'set-secret-key' with pid 11
2016-09-10 23:45:19,239 INFO spawned: 'app-uwsgi' with pid 12
2016-09-10 23:45:19,241 INFO spawned: 'start-mysql' with pid 13
2016-09-10 23:45:19,272 INFO exited: set-secret-key (exit status 0; not expected)
2016-09-10 23:45:19,330 CRIT reaped unknown pid 72)
2016-09-10 23:45:19,408 CRIT reaped unknown pid 122)
2016-09-10 23:45:19,505 CRIT reaped unknown pid 231)
2016-09-10 23:45:19,513 CRIT reaped unknown pid 256)
2016-09-10 23:45:19,536 CRIT reaped unknown pid 306)
2016-09-10 23:45:19,537 CRIT reaped unknown pid 307)
2016-09-10 23:45:19,565 CRIT reaped unknown pid 333)
2016-09-10 23:45:19,566 CRIT reaped unknown pid 340)
2016-09-10 23:45:19,616 CRIT reaped unknown pid 354)
2016-09-10 23:45:19,647 CRIT reaped unknown pid 363)
2016-09-10 23:45:20,250 INFO success: celery-app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-09-10 23:45:20,250 INFO success: nginx-app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-09-10 23:45:20,251 INFO success: rabbitmq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-09-10 23:45:20,251 INFO success: app-uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-09-10 23:45:20,251 INFO success: start-mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-09-10 23:45:20,293 INFO spawned: 'set-secret-key' with pid 515
2016-09-10 23:45:20,317 INFO exited: set-secret-key (exit status 0; not expected)
2016-09-10 23:45:20,524 CRIT reaped unknown pid 539)
2016-09-10 23:45:20,529 CRIT reaped unknown pid 545)
2016-09-10 23:45:20,587 CRIT reaped unknown pid 546)
2016-09-10 23:45:20,587 CRIT reaped unknown pid 586)
2016-09-10 23:45:22,351 INFO spawned: 'set-secret-key' with pid 653
2016-09-10 23:45:22,360 INFO exited: set-secret-key (exit status 0; not expected)
2016-09-10 23:45:25,365 INFO spawned: 'set-secret-key' with pid 682
2016-09-10 23:45:25,378 INFO exited: set-secret-key (exit status 0; not expected)
2016-09-10 23:45:26,364 INFO gave up: set-secret-key entered FATAL state, too many start retries too quickly
2016-09-10 23:45:27,799 CRIT reaped unknown pid 397)
2016-09-10 23:45:27,820 INFO exited: rabbitmq (exit status 0; expected)
2016-09-10 23:45:27,992 INFO exited: start-mysql (exit status 0; expected)
2016-09-10 23:45:28,151 CRIT reaped unknown pid 616)

CTG Not Valid Start Codon

Hi all,

In my database some of the genomes use CTG as their start codon. This appears to not be a valid start codon in PhamDB. Is there any way I can edit the parameters of acceptable start codons to include CTG so I can add these phages to the database?

Thanks!