kanakb / pyhelix

Python participant and spectator for Apache Helix

pyhelix

Python bindings for Apache Helix

Install:

sudo easy_install pyhelix

OR

sudo pip install pyhelix

Start a participant:

# Imports
import pyhelix.participant as participant
import pyhelix.examples.dummy.dummy_statemodel as statemodel

# Define a participant
p = participant.Participant('test-cluster', 'myhost', 12120, 'zkhost:2181')

# Register a state model factory
p.register_state_model_fty('MasterSlave', statemodel.DummyStateModelFactory())

# Connect
p.connect()

# Disconnect
p.disconnect()

pyhelix's Issues

Support batch messaging mode

This was a feature added in 0.6.1, and it would be nice to support it as it can reduce the number of callbacks.

Clean old current state on reconnect

If we have to reconnect, the current state from the previous session should be removed

Need a reset in the state model

If a session is lost, we should have a method in state model that is invoked so that users can clean up.

Process transitions faster

For resources with a large number of assigned partitions, this participant can be slow in processing everything. There are a few ways to tackle this:

• batch ZK operations
• make some ZK operations asynchronous
• native code?

Forced participant ID convention: "host_port"?

This formatting is hardcoded here:

class Participant(object):
    ...
    self._participant_id = '{0}_{1}'.format(host, port)

But there's much more flexibility in naming instances when using helix-admin.sh to set up the cluster initially (maybe the setup flow if auto-join not allowed for some cluster). Should Participant also have this same level of flexibility?

Write a more compelling example

One possibility is a distributed message queue system, similar to the RabbitMQ recipe in the Helix Java source.

Write more unit tests

Automatically set current state in the state model

https://github.com/kanakb/pyhelix/blob/master/pyhelix/statemodel.py has a get_current_state, but leaves it up to the user to update it. pyhelix should do this automatically

Handle client exceptions

We should be going into ERROR state when the participant code throws exceptions.

Pre-connect callbacks

Need a way to register a callback before the connection is established

Write a helix agent

Similar to the one in Java, this should invoke commands based on state transitions while monitoring a process whose liveness indicates whether or not the connection should remain open.

Have a default transition handler

We shouldn't fail if the user doesn't support a transition. We should log an error and move on.

Make sure marking messages as read is race-free

There's a potential race condition between marking messages as read, and deleting them post-transition.

Increase executor parallelism

Right now this is 1, and it really should be configurable

Handle change in ZK connection state

More instructions here: http://kazoo.readthedocs.org/en/latest/basic_usage.html

Write a basic spectator

Need this to write pure-Python apps.

Disable partition/node on ERROR-DROPPED

Support Python 3.x

The version number and other release constants should be in one place

Handle client timeouts

If timeouts are enabled, we should go into ERROR state for transitions that take too long.

Write integration tests

The kazoo.testing module has a usable test harness for this purpose.

Support property store

All that is needed is to add a key that takes arbitrary number of arguments under the PROPERTYSTORE path