man-group / mdf

Data-flow programming toolkit for Python

mdf

mdf stands for man systematic strategies data-flow. mdf is a toolkit for expressing programs in Python as directed acyclic graphs (DAGS). mdf was developed to make it easier to wire together quantitative trading strategies composed of related computations over time-series data, but it can be used as a toolkit for describing general computations as DAGS.

The full documentation, with examples, can be found here: http://mdf-data-flow-programming-toolkit.readthedocs.org/en/latest/

mdf's Issues

Node serialization fails with Pyro >= 2.20

Tests involving serializing nodes fail on the newer pyro versions:

Traceback (most recent call last):
  File "/home/travis/build/SleepingPills/mdf/mdf/tests/test_regression.py", line 37, in test_df_differ
    diffs = mdf.regression.run(date_range, [df_differ], lhs=None, rhs=None)
  File "/home/travis/build/SleepingPills/mdf/mdf/regression/__init__.py", line 281, in run
    rhs_modulenames=rhs_modulenames)
  File "/home/travis/build/SleepingPills/mdf/mdf/regression/__init__.py", line 239, in get_contexts
    lhs_ctx = lhs_server.get_remote_context(ctx)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/Pyro4/core.py", line 160, in __call__
    return self.__send(self.__name, args, kwargs)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/Pyro4/core.py", line 290, in _pyroInvoke
    compress=Pyro4.config.COMPRESSION)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/Pyro4/util.py", line 159, in serializeCall
    data=self.dumpsCall(obj, method, vargs, kwargs)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/Pyro4/util.py", line 392, in dumpsCall
    return serpent.dumps((obj, method, vargs, kwargs))
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 62, in dumps
    return Serializer(indent, set_literals).serialize(obj)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 211, in serialize
    self._serialize(obj, out, 0)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 240, in _serialize
    getattr(self, method_name, self.ser_default_class)(obj, out, level)  # dispatch
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 290, in ser_builtins_tuple
    self._serialize(elt, out, level + 1)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 240, in _serialize
    getattr(self, method_name, self.ser_default_class)(obj, out, level)  # dispatch
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 290, in ser_builtins_tuple
    self._serialize(elt, out, level + 1)
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 240, in _serialize
    getattr(self, method_name, self.ser_default_class)(obj, out, level)  # dispatch
  File "/home/travis/virtualenv/python2.6/lib/python2.6/site-packages/serpent.py", line 436, in ser_default_class
    raise TypeError("don't know how to serialize class " + str(obj.__class__) + ". Give it vars() or an appropriate __getstate__")
TypeError: don't know how to serialize class <type 'mdf.context.MDFContext'>. Give it vars() or an appropriate __getstate__

Unit tests test_regression and test_remote fork bomb when run with nose or py.test

Class inheritance causes duplicate evaluation of nodes

When using class inheritance for classes that group together nodes, a node within the base class will be evaluated separately in each base and derived class that it is accessed from.

from mss.mdf import evalnode
from mss.mdf import MDFContext

class Base(object):
    @evalnode
    def expensive_node(cls):
        print "Running expensive calculation."

class Derived(Base):
    @evalnode
    def derived_node_that_uses_expensive_node(cls):
        cls.expensive_node()

ctx = MDFContext()

# expensive_node is evaluated once within Base...
ctx[Base.expensive_node]
# [Output] Running expensive calculation.
ctx[Base.expensive_node]
ctx[Base.expensive_node]

# ...and again within Derived.
ctx[Derived.derived_node_that_uses_expensive_node]
# [Output] Running expensive calculation.

# (Just the once)
ctx[Derived.expensive_node]

Is it possible for the base and derived classes to share the same node within the context (so long as the node is not overridden, obviously)?

does MDFViewerFrame work on OSX?

first at all - many thanks for making MDF available in open-source. awesome work - this is super cool.

I just started playing with it and most of the functionality works very nicely apart from the graph debugger.

I am running mdf on OSX 10.9.5. The example in %mdf_help breaks with the following stacktrace.

NameError: global name 'viewer' is not defined

if I do "from mdf import viewer" import complains about missing pywintypes.

/Library/Python/2.7/site-packages/mdf-2.2.1-py2.7-macosx-10.9-intel.egg/mdf/viewer/init.py in ()
10 """
11 import wx
---> 12 from frame import MDFViewerFrame
13 from ..context import _get_current_context
14 from ..nodes import MDFNode

/Library/Python/2.7/site-packages/mdf-2.2.1-py2.7-macosx-10.9-intel.egg/mdf/viewer/frame.py in ()
19 from ..builders import DataFrameBuilder
20 from ..nodes import MDFVarNode
---> 21 import excel
22 import traceback
23

/Library/Python/2.7/site-packages/mdf-2.2.1-py2.7-macosx-10.9-intel.egg/mdf/viewer/excel.py in ()
4 from decorator import decorator
5 import numpy as np
----> 6 import pywintypes
7 import re
8

ImportError: No module named pywintypes

so looks like mdf is not portable across multiple platforms or am I doing something stupid?

many thanks,
Regards, Roman

Switch to MIT License

mdf.lab docs need a tidy up

• Code referring to mdf.pylab

error with python 3.5 multiprocessing.util

In python 3.5, I get an error running any graph, from line 304 in mdf.runner: multiprocessing.util.active_children()

I'm guessing that either this has been moved in a newer version of python or I'm doing something silly involving main

"evalnodes can only be accessed via the class, not instance"

Hi there - starting my journey in using MDF. A major question I have is... why evalnodes can only be accessed via the class, not instance?

Specifically, here's the dummy example I am trying to construct.

class Fund(object):
    data_source = varnode(default='isin')
    data_source_id = varnode()

    @evalnode()
    def __str__(self):
        return '{} ({}: {})'.format(self.long_name, self.data_source, self.data_source_id)

    @evalnode()
    def long_name(self):
        return self.row_getter()['long_name']

    @evalnode()
    def mgmt_fee(self):
        return self.row_getter()['mgmt_fee']

    @evalnode()
    def row_getter(self):
        return self.fund_ref_data_by_isin()[self.data_source_id()]

    @evalnode()
    def fund_ref_data_by_isin(self):
        df = pd.read_excel('data.xlsx', 'funds', index_col='data_source_id')
        return df.to_dict(orient='index')

'data.xlsx' basically has a flat table key by the ISIN and bunch of reference data.

Essentially I want these to work within the same context

ctx = MDFContext()
fund1 = Fund()
fund2 = Fund()

ctx[fund1.data_source_id] = 'IE00B12V2W34'
ctx[fund2.data_source_id] = 'IE00B1XK9C88'
print(ctx[fund1.long_name])
print(ctx[fund2.long_name])

Right now I hit the error
AssertionError: evalnodes can only be accessed via the class, not instance

But if we can't apply MDF into instance.... then what would be the recommended way to achieve my use case above? I can't imagine having to create 2 context or to have to toggle them around?

Also.... trying to see if there are any larger example code of MDF. It's actually an amazing tool in modelling dependency (conceptually it's really like Excel but better). So would be great to see some large scale example....

mdf run() fails to incorporate tzinfo when passed a pandas.DateRange

pandas.DateRange is deprecated. If run() is passed a date_range parameter constructed from pandas.DateRange it will actually be of time DatetimeIndex. Consequently run() fails to extract tzinfo from date_range since it checks to see if date_range has type pandas.DateRange first. The workaround is to pass in a tzinfo parameter explicitly to run(). The fix should be to remove the dependence on pandas.DateRange.

still used/best choice

Just raising an issue to ask if this is still in use and still makes sense to use, sorry to bother

upload to pypi

apparently nobody owns mdf, so i'm name squatting for you:
https://pypi.org/project/mdf/

Let me know who to transfer to.

Vectorization and Parakeet/Numba support

Add more support for vectorized operation, perhaps auto-vectorization and integration with Parakeet/Numba for dramatic speed improvements.

Auto-parallelized graph reduction support

Set up travis

As it says on the tin.

Ideally, we'll have 2.6, 2.7 and 3.3 CI builds running.

Restore IPython magic for mdf

The IPython magic was omitted from the initial open-sourcing. It will be added back in a new package called mdf.lab

Add more information to readme file

A quick and simple intro on what MDF is, what problems it solves and some simple examples. Also, a short explanation for the name would be useful.

python exits with 255 exit code on ctx[none] = 55

ctx = MDFContext()
ctx[None] = 55

results in runtime terminating; it would be better to throw: easier to diagnose the underlying issue (node used in indexer being None)