toros-astro / corral Goto Github PK

The Powerful Pipeline Framework

License: BSD 3-Clause "New" or "Revised" License

Python 54.95% Mako 0.19% TeX 44.85%

python python3 framework pipeline astronomy database data-science oop

corral's Introduction

The Powerful Pipeline Framework

Corral will solve your pipeline needs by merging a database full connection interface with a MVC model, by making you able of editing your custom schemas and adding the possibility of writting specific processing steps following a intuitive data handling model.

Features

Command line interface with standard utilities
Editable commands on your application
Completely built on SQLAlchemy ORM
Allows the introduction of Steps
Follows the MVC (Model View Controller) pattern

Support

Full Documentation and Tutorial: http://corral.readthedocs.io/
Community Forum: https://groups.google.com/forum/#!forum/corral-users-forum
Issues: If you have issues please report them as a issue here: https://github.com/toros-astro/corral/issues

User Installation

The easiest way to install corral is using pip

$ pip install -U corral-pipeline

Developmment

Install project by cloning from Corral github:

$ git clone https://github.com/toros-astro/corral.git

and by making pip install -e ., or the classic python setup.py install.

You can also run the install by giving the github link directly to pip:

$ pip install -e git+https://github.com/toros-astro/corral.git

License

BSD-3: https://opensource.org/licenses/BSD-3-Clause
BSD-3 Licence Explained: https://tldrlegal.com/license/bsd-3-clause-license-(revised)

corral's People

Contributors

Stargazers

Watchers

Forkers

mardom edwardbetts silky nudomarinero bxy8804 neutrinoceros

corral's Issues

requirements

Didin't we had requirements already???

Anyway. Please @leliel12 run a pip freeze for me.

Unittest

Create some unittests for run inside tox #2

Astropy Affiliated Package Review

With apologies for the delay, this package has been reviewed by the Astropy coordination committee in relation to the Astropy affiliated package ecosystem.

We have adopted a review process for affiliated package that includes assigning quantitative ‘scores’ (red/orange/green) for different categories of review. You can read up more about this process here. (This document, currently in Google Docs, will be moved to the documentation in the near future.) For each of the categories below we have listed the score and have included some comments when the score is not green.

Functionality/Scope
No further comments
Integration with Astropy ecosystem
At the moment there is no mention of Astropy in the corral documentation. Would it be possible to include a tutorial that uses Astropy somehow, for example to process FITS files? I think this will make it more directly useful to Astronomers
Documentation
Most functions and classes I checked in the code didn't have any docstrings - making it harder for someone to jump into the code to contribute. It would be good to document the code better, and to add an API docs page to the documentation.
Testing
No further comments
Development status
No further comments
Python 3 compatibility
No further comments

Summary/Decision: Things are looking good! There are a few areas where things could be improved as described above. However, these aren't critical and this package meets the criteria to be an Astropy affiliated package, so we are happy to accept it.

To finalize the acceptance, please open a pull request to add an entry to the registry.json file for this package in the website repository.

If you agree with the above review, please feel free to close this issue. If you have any follow-up questions or disagree with any of the comments above, leave a comment and we can discuss it here. At any point in future you can request a re-review of the package if you believe any of the scores should be updated - contact the coordination committee, and we’ll do a new review.

Finally, please keep the title of this issue as-is (“Astropy Affiliated Package Review”) to make it easy to search for affiliated package reviews in future.

including core tables in models.py causes corral to fail

I think this is a bug, if instead of class models I include a not ORM table for the many to many it causes corral to fail.

Adding these tables

calib_combination = db.table(
    'calib_combination', db.Model.metadata,
    db.Column('calib_id', db.ForeignKey('calibfile.id'),
              primary_key=True),
    db.Column('combination_id', db.ForeignKey('combination.id'),
              primary_key=True))


paw_stack = db.table('paw_stack', db.Model.metadata,
                     db.Column('paw_id', db.ForeignKey('pawprint.id'),
                               primary_key=True),
                     db.Column('stack_id', db.ForeignKey('stack.id'),
                               primary_key=True))

results in this traceback:

$ corraldo createdb
/Users/utb/.virtualenvs/corral/lib/python2.7/site-packages/flask/exthook.py:71: ExtDeprecationWarning: Importing flask.ext.babel is deprecated, use flask_babel instead.
  .format(x=modname), ExtDeprecationWarning
Traceback (most recent call last):
  File "in_corral.py", line 10, in <module>
    cli.run_from_command_line()
  File "/Users/utb/Academia/Research/TOROS/Code/corral/corral/cli/__init__.py", line 157, in run_from_command_line
    parser = create_parser()
  File "/Users/utb/Academia/Research/TOROS/Code/corral/corral/cli/__init__.py", line 128, in create_parser
    load_commands_module()
  File "/Users/utb/Academia/Research/TOROS/Code/corral/corral/cli/__init__.py", line 119, in load_commands_module
    return util.dimport(COMMANDS_MODULE)
  File "/Users/utb/Academia/Research/TOROS/Code/corral/corral/util.py", line 40, in dimport
    return importlib.import_module(importpath)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/utb/Academia/Research/TOROS/Code/toritos_corral/toritos/commands.py", line 3, in <module>
    import models
  File "/Users/utb/Academia/Research/TOROS/Code/toritos_corral/toritos/models.py", line 290, in <module>
    primary_key=True))
  File "<string>", line 2, in table
  File "/Users/utb/.virtualenvs/corral/lib/python2.7/site-packages/sqlalchemy/sql/selectable.py", line 1334, in __init__
    self.append_column(c)
  File "/Users/utb/.virtualenvs/corral/lib/python2.7/site-packages/sqlalchemy/sql/selectable.py", line 1347, in append_column
    self._columns[c.key] = c
AttributeError: 'MetaData' object has no attribute 'key'

Translate Tutorial 1

http://corral.readthedocs.io/en/latest/intro/tuto01.html

the sources are in corral/docs/sources/intro/tutoXX.rst

fix CLI tutorial

Foreign Keys

In the foreign Keys assignation there are bad syntax lines.

in-corral crashes in clean pull from repo

I just pulled a fresh copy of the repo and tried

python in-corral.py

with no commands and it crashes like this.

Traceback (most recent call last):
  File "in_corral.py", line 10, in <module>
    from corral import cli
  File "/Users/utb/Academia/Research/TOROS/Code/corral_test/corral/corral/cli/__init__.py", line 9, in <module>
    from .. import core, conf, util, exceptions
  File "/Users/utb/Academia/Research/TOROS/Code/corral_test/corral/corral/core.py", line 7, in <module>
    from . import db
  File "/Users/utb/Academia/Research/TOROS/Code/corral_test/corral/corral/db.py", line 16, in <module>
    from . import conf, util
  File "/Users/utb/Academia/Research/TOROS/Code/corral_test/corral/corral/conf.py", line 73, in <module>
    settings = LazySettings(CORRAL_SETTINGS_MODULE)
  File "/Users/utb/Academia/Research/TOROS/Code/corral_test/corral/corral/conf.py", line 35, in __init__
    self._settings = util.dimport(settings_module_name)
  File "/Users/utb/Academia/Research/TOROS/Code/corral_test/corral/corral/util.py", line 43, in dimport
    raise ImportError("No module named {}".format(importpath))
ImportError: No module named toritos.settings

Can anyone reproduce? Maybe it's just me. I'll look into it in the meantime.

create tutorial for tests

Database Manual Test

Populate te toritos models with some fake data.
Write some queries to recover the values

Calibration files

Calibration files must be stored in DB as calfile and not as pawprint.
Things are different from this approach.

Things to change:

load must change behaviour with imagetypes,

or:

from raw data in pawprint a new branch should open for calib files.

finally:

Steps for calibration must change completely since now querys should go against another tables.

translate tuto 5

ya mañana agrego el tuto 6 🍡

`run_style` from QA uses deprecated API from Flake8

Tests fail for test_run_style

This is from running tox -r -e py310 in branch portpy3.10

======================================================================
ERROR: test_run_style (tests.test_qa.TestRunStyle)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bruno/Devel/toros-astro/corral/tests/test_qa.py", line 169, in test_run_style
    report, text = qa.run_style()
  File "/home/bruno/Devel/toros-astro/corral/corral/qa.py", line 590, in run_style
    report = pep8.check_files()
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/flake8/engine.py", line 181, in check_files
    return self._retry_serial(self._styleguide.check_files, paths=paths)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/flake8/engine.py", line 172, in _retry_serial
    return func(*args, **kwargs)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pycodestyle.py", line 1875, in check_files
    self.input_dir(path)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pycodestyle.py", line 1911, in input_dir
    runner(os.path.join(root, filename))
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/flake8/engine.py", line 126, in input_file
    return fchecker.check_all(expected=expected, line_offset=line_offset)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pycodestyle.py", line 1608, in check_all
    self.check_ast()
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pycodestyle.py", line 1554, in check_ast
    checker = cls(tree, self.filename)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/flake8/_pyflakes.py", line 62, in __init__
    super(FlakesChecker, self).__init__(tree, filename,
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pyflakes/checker.py", line 425, in __init__
    self.handleChildren(tree)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pyflakes/checker.py", line 747, in handleChildren
    self.handleNode(node, tree)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pyflakes/checker.py", line 794, in handleNode
    handler(node)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pyflakes/checker.py", line 747, in handleChildren
    self.handleNode(node, tree)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pyflakes/checker.py", line 793, in handleNode
    handler = self.getNodeHandler(node.__class__)
  File "/home/bruno/Devel/toros-astro/corral/.tox/py310/lib/python3.10/site-packages/pyflakes/checker.py", line 627, in getNodeHandler
    self._nodeHandlers[node_class] = handler = getattr(self, nodeType)
AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'

----------------------------------------------------------------------
Ran 124 tests in 3.329s

FAILED (errors=1)
ERROR: InvocationError for command /home/bruno/Devel/toros-astro/corral/.tox/py310/bin/python run_tests.py -v (exited with code 1)
______________________________________________________________________ summary ______________________________________________________________________
ERROR:   py310: commands failed

add tox

integrate with tox test enviroment

Migrations

Integrate corral with alembic
Use a db migrations folder your_project/migrations
Try to create 2 commands python in_corral.py makemigrations and python in_corral.py migrate with the same functionalities of django migrations
Map all subcommands of alembic with command python in_corral.py alembic <alembic_sub_cms>

Separate corrral from toritos

This will be happend soon

Translate tutp 2

http://corral.readthedocs.io/en/latest/intro/tuto02.html

the sources are in corral/docs/sources/intro/tutoXX.rst

Add test utility

support for create a command like

$ python in_corral.py test

The command must configure an environment with a database specified in the settings.TEST_CONNECTION
The test must libe in <your_project>/tests.py
The core functionality must live in corral/qa.py

Running StepDarkPreprocess

While running StepDarkPreprocess:

a wild bug appears...

What would ASH like to do?

~$ python in_corral.py --stacktrace run --sync --step StepDarkPreprocess

[Toritos-INFO @ 2015-12-06 17:21:29,214] ROLLBACK
Class '__builtin__.tuple' is not mapped
---------------------------------------------------------------------------
UnmappedInstanceError                     Traceback (most recent call last)
/home/bruno/.virtualenvs/corral/src/corral/in_corral.py in <module>()
      8 
      9     from corral import cli
---> 10     cli.run_from_command_line()

/home/bruno/.virtualenvs/corral/src/corral/corral/cli/__init__.pyc in run_from_command_line()
     69         sys.stderr.write("{}\n".format(err))
     70         if gkwargs.get("stacktrace"):
---> 71             six.reraise(type(err), err, six.sys.exc_traceback)

/home/bruno/.virtualenvs/corral/src/corral/corral/cli/__init__.pyc in run_from_command_line()
     65     func, kwargs, gkwargs = parser.parse_args(sys.argv[1:])
     66     try:
---> 67         func(**kwargs)
     68     except BaseException as err:
     69         sys.stderr.write("{}\n".format(err))

/home/bruno/.virtualenvs/corral/src/corral/corral/cli/commands.pyc in handle(self, step_classes, sync)
    205         procs = []
    206         for step_cls in step_classes:
--> 207             proc = run.execute_step(step_cls, sync=sync)
    208             procs.append(proc)
    209         if not sync:

/home/bruno/.virtualenvs/corral/src/corral/corral/run/step.pyc in execute_step(step_cls, sync)
     92     runner.set_target(step_cls)
     93     if sync:
---> 94         runner.run()
     95     else:
     96         runner.start()

/home/bruno/.virtualenvs/corral/src/corral/corral/run/step.pyc in run(self)
     29                 step.validate(obj)
     30                 generator = step.process(obj) or []
---> 31                 if not hasattr(generator, "__iter__"):
     32                     generator = (generator,)
     33                 for proc_obj in generator:

/home/bruno/.virtualenvs/corral/src/corral/corral/run/base.pyc in save(self, obj)
     41 
     42     def save(self, obj):
---> 43         self.session.add(obj)
     44 
     45     @abc.abstractmethod

/home/bruno/.virtualenvs/corral/local/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in add(self, instance, _warn)
   1573             state = attributes.instance_state(instance)
   1574         except exc.NO_STATE:
-> 1575             raise exc.UnmappedInstanceError(instance)
   1576 
   1577         self._save_or_update_state(state)

UnmappedInstanceError: Class '__builtin__.tuple' is not mapped

Import exception in util/dimport.py misleading message

The try/except in util/dimport.py raises an error if a module can't be loaded.
The raise message should be something like "Problem loading module XXX", instead of ""No module named XXX" as it is right now.
As an example, the toritos.settings module fails to load if numpy dependency is not installed, even though the toritos.settings module exists.

stackstate = db.relationship(
    "StackState", backref=db.backref('stack_statechanges', order_by=id))

make look like

stackstate_id = Column(Integer, ForeignKey('stackstate.id'))
stackstate = db.relationship(
        "StackState", backref=db.backref('stack_statechanges', order_by=id))