zvibaratz / pylabber

A database manager for neuroscience

🧠 Neuroscience PhD student at Tel-Aviv University

⭐ Currently working on The Labbing Project alongside my PhD thesis in Prof. Yaniv Assaf's lab

💬 Ask me about MRI data management, Python and neuroimaging, Linux, open source software, open science, home pickling

        

pylabber's Issues

Allow remote access to the application

Learn how to expose a port and allow people to connect to the application through the server.

Add SSL

Add SSL for secure connectivity.

drop coreapi dependency

(venv) sivan@titan:~/pylabber/pylabber$ ./manage.py generateschema
Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/sivan/pylabber/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/sivan/pylabber/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/sivan/pylabber/venv/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/sivan/pylabber/venv/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "/home/sivan/pylabber/venv/lib/python3.6/site-packages/rest_framework/management/commands/generateschema.py", line 20, in handle
    assert coreapi is not None, 'coreapi must be installed.'
AssertionError: coreapi must be installed.

feature: need to be able to download a subject's MRI data

-1st Iteration: Download per subject returned as a zip file of all of the scans - basically to zip the store's existing directory structure.
-2st: Iteration: Download all data of Study "cortical layers", all scans that one of the groups that is associated is as the selected study will be added to the resulting zip archive.

Create custom LabMembership model

This model should store the affiliation of the researcher to the lab, i.e. PhD candidate, contractor, lab manager, etc.

remove all celery references

(venv) sivan@titan:~/pylabber/pylabber$ git branch
  localdev
  master
* study_groups
(venv) sivan@titan:~/pylabber/pylabber$ grep -Hir celery *
build/lib/pylabber/celery.py:from celery import Celery
build/lib/pylabber/celery.py:app = Celery("pylabber")
build/lib/pylabber/celery.py:app.config_from_object("django.conf:settings", namespace="CELERY")
build/lib/pylabber/__init__.py:# import celery
build/lib/pylabber/__init__.py:from .celery import app as celery_app
build/lib/pylabber/__init__.py:__all__ = ("celery_app",)
Binary file dist/pylabber-0.1.0-py3.6.egg matches
Binary file pylabber/__pycache__/__init__.cpython-36.pyc matches
Binary file pylabber/__pycache__/celery.cpython-36.pyc matches
pylabber/celery.py:from celery import Celery
pylabber/celery.py:app = Celery("pylabber")
pylabber/celery.py:app.config_from_object("django.conf:settings", namespace="CELERY")
pylabber/__init__.py:# import celery
pylabber/__init__.py:from .celery import app as celery_app
pylabber/__init__.py:__all__ = ("celery_app",)
pylabber.egg-info/SOURCES.txt:pylabber/celery.py
requirements/requirements.txt:celery==4.3.0
requirements/requirements.txt:django-celery-beat==1.4.0

Create an AMI for instance creation automation

In order to automate the creation of instances, we should create an appropriate AMI.

Add DICOM series conversion to NIfTI

The interface should allow configuring the location of dcm2nii or some other conversion program and using it on a DICOM series when needed for an analysis or requested by a user (there should be some kind of button or configuration that creates and keeps a reference for a NIfTI version).

Association of data with study

Views and templates for the association of data with a particular study need to be created.

Figure out best way to sync with a remote windows share (SMB path)

Either create a local directory the mirrors the remote directory (wastes a lot of storage space) or find a way to quickly filter known files and only copy new ones.

Code review and refactoring

General code review and refactoring before adding more functionality and creating more associated apps.

Anonymize viewed data

Browsable data should not include any compromising information, at least for non-admin users.

Testing

Writing tests for current application functionality.

Automate the flow of new data into the database

Once new DICOM data is copied from a remote location, it should be added to the database and archived accordingly.

feature: User specific data access via filtering.

A researchers should have access to Subjets & Scans that he was only been granted access to do - and this should be manifested in the data view.

https://stackoverflow.com/questions/19707237/use-get-queryset-method-or-set-queryset-variable

Support for direct dcm4chee integration.

We should use pynetdicom to integrate with dcm4chee properly.

Figure out best way to sync with a remote windows share (SMB path)

Preferably find a quick way to spot new files and copy only them.

Add script to create an EC2 instance programmatically

This should include the creation of a dedicated security group and key pair.

Anonymized data associations

Figure out best way to maintain a secure key between personal information and anonymized medical information

Documentation

Write documentation and usage examples for existing code.

Nipype integration

Nipype should be integrated, possibly by creating another separate application that will constitute a django-based UI. This app should include views and templates for analysis creation and management.

Deploy on AWS

Create an EC2 instance to serve the app as well as an associated RDS using psql and S3 bucket.
We also need load balancing and scaling groups put in place.

clean up all committed migrations and migration code from the code base

I have realized where the problem with recurrent migrations we have: They are committed into the GIT tree. We need to clean this up. The only thing that should be pushed is plain code nothing else.

Create a form for SMB location configuration

Credentials must be safely serialized (as env variables?).

Embedding Jupyter notebooks?

In order to allow for easy data exploration, it might be best to simply enable embedding Jupyter notebooks in the study/data/results DetailViews. This StackOverflow response will probably be helpful.

Questionnaire model

Create a questionnaire model and associate it with a Google Forms web address. Add methods for data update and summary.

Pipe and Pipeline models

A pipe should reference a script and manage input and output validation. A pipeline should be some predetermined ordered combination of pipes. Not sure yet how the status/result of each pipe should be stored for future reference.

migrations are broken on headless

Traceback (most recent call last):
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/apps/config.py", line 178, in get_model
    return self.models[model_name.lower()]
KeyError: 'group'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 15, in <module>
    execute_from_command_line(sys.argv)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute
    self.check()
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check
    include_deployment_checks=include_deployment_checks,
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 65, in _run_checks
    issues.extend(super()._run_checks(**kwargs))
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/management/base.py", line 377, in _run_checks
    return checks.run_checks(**kwargs)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/urls/resolvers.py", line 398, in check
    for pattern in self.url_patterns:
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/urls/resolvers.py", line 579, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/urls/resolvers.py", line 572, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/pylabber/pylabber/urls.py", line 36, in <module>
    path("api/", include("django_mri.urls", namespace="mri")),
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/urls/conf.py", line 34, in include
    urlconf_module = import_module(urlconf_module)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django_mri/urls.py", line 2, in <module>
    from django_mri import views
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django_mri/views.py", line 18, in <module>
    from django_mri.serializers import (
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django_mri/serializers.py", line 11, in <module>
    Group = apps.get_model(app_label=group_app_label, model_name=group_model_name)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/apps/registry.py", line 210, in get_model
    return app_config.get_model(model_name, require_ready=require_ready)
  File "/home/ubuntu/pylabber/venv/lib/python3.6/site-packages/django/apps/config.py", line 181, in get_model
    "App '%s' doesn't have a '%s' model." % (self.label, model_name))
LookupError: App 'research' doesn't have a 'Group' model.
r

can you please fix ? :)