https://en.wikipedia.org/wiki/Bagua
Xun is a distributed and functional Python framework for cluster compute. Rather than focusing on batching jobs, xun is about defining values declaratively.
Standalone example xun project file for computing fibonacci numbers
import xun
@xun.function()
def fibonacci_number(n):
return f_n_1 + f_n_2
with ...:
f_n_1 = (
0 if n == 0 else
1 if n == 1 else
fibonacci_number(n - 1)
)
f_n_2 = fibonacci_number(n - 2) if n > 1 else 0
@xun.function()
def fibonacci_sequence(n):
return sequence
with ...:
sequence = [fibonacci_number(i) for i in range(n)]
def main():
"""
Compute and print the first 10 fibonacci numbers
"""
blueprint = fibonacci_sequence.blueprint(10)
sequence = blueprint.run(
driver=xun.functions.driver.Sequential(),
store=xun.functions.store.Memory(),
)
for num in sequence:
print(num)
if __name__ == '__main__':
main()To see a visualization of the call graph:
xun graph examples/fibonacci.py "fibonacci_sequence(10)"Let's break down the code from fibonacci_number in the example above in to 4 parts
@xun.function()The decorator @xun.function() compiles this function into a xun function. Xun functions are functions that are meant to be executed in parallel, possibly on remote workers.
def fibonacci_number(n):The function definition is just a normal python function definition.
return f_n_1 + f_n_2The body of the function is just regular python, it has as expected access to the function arguments, but it also has access to the variables defined in the special xun definitions statement.
with ...:
f_n_1 = (
0 if n == 0 else
1 if n == 1 else
fibonacci_number(n - 1)
)
f_n_2 = fibonacci_number(n - 2) if n > 1 else 0Statements on the form with ...: we refer to as xun definitions. They introduce new syntax and rules that we will get more into in the next section. Note for example that the recursive calls to fibonacci_number(n) are memoized in the context store and can therefore, after scheduling, be run in parallel.
In fact, xun works by first figuring out all the calls that will happen, building a call graph, and scheduling the calls such that any previous call that we may depend on is executed before we evaluate the current call. This requires the call graph to be a directed acyclic graph (DAG).
@xun.function()
def do_some_work(some_values):
result = expensive_computation(data)
with ...:
data = depencency(fixed_values)
fixed_values = [fix(v) for v in some_values]In the above example, a job takes in some iterable some_values as argument, polishes the values in it and calls another context function that it depends on. Note that the order of the statements inside the xun defintions statements does not matter. The syntax of xun definitions statements is similar to where clauses in Haskell and has rules that differ from standard python. In general, for xun definitions statements the following apply:
- Order of statements is arbitrary
- Xun functions can only be called from xun definition statements (
with ...:) - Only assignments and free expressions are allowed
- There can only be one xun definitions statement per xun function
- Values cannot be modified
- If a function modifies a value passed to it, the changes will not be reflected for the value in the definitions. That is, arguments to calls are passed by value.
- Any code in xun definitions statements will be executed during scheduling, so the heavy lifting should be done in the function body, and not inside the xun definitions statements
Xun definition statements allow xun to figure out the order of calls needed to execute a xun program.
As calls to xun functions are executed and finished, the results are saved in the store of the context. Stores are classes that satisfy the requirements of collections.abc.MutableMapping, are pickleable, and whos state is shared between all instances. Stores can be defined by users by defining a new class with extending xun.functions.store.Store.
Drivers are the classes that have the responsibility of executing programs. This includes scheduling the calls of the call graph and managing any concurrency.
from math import radians
import numpy as np
def not_installed():
pass
@xun.make_shared
def not_installed_but_shared():
pass
@xun.function()
def xun_function():
not_installed() # Not OK
not_installed_but_shared() # OK
radians(180) # OK because the function is builtin
np.array([1, 2, 3]) # OK because the function is defined in an installed moduleBecause xun functions are pickled, any function they reference must either be installed on the system or be represented differently. xun comes with a decorator, @xun.make_shared, that can make many functions serializable.
- You should only reference global scope from a function that would not change the outcome of the function if changed. Global scope is not considered when identifying results, thus changes to the global scope might give undesired results.
- A good use for global scope is to specify configuration values, such as cluster addresses or file system paths.
data_dir = '/path/to/data' @xun.function() def load_data(): return load(data_dir)
- If you want a variable change to impact your results, define it as a xun function. For example to configure a simulation with a fixed seed, define the value as a xun function rather than a variable in global scope
@xun.function() def simulation_seed(): return 196883
Functions can yield auxiliary results accessible through interfaces. This is useful if a function returns large, but separable results. Results can be yielded to interfaces specified by a decorator @<xun.Function>.interface. This let's the original function write results accessible through the interface as if they were xun functions. Yields from a xun function are declared in the function body as yield statements of the form yield <call-to-interface> is <expr>.
Interfaces must specify which function that should be responsible for producing it's result.
In this example the function f returns what is passed to it, but in addition yields results to interfaces even and odd. Calling even and odd interfaces will return the n-th even and odd integer respectively.
import xun
@xun.function()
def f(n):
yield even(n) is n * 2
yield odd(n) is n * 2 + 1
return n
@f.interface
def even(n):
yield from f(n)
@f.interface
def odd(n):
yield from f(n)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent