richardknop / machinery Goto Github PK

When my function is called and it returns an error, it is converted to string, rather than an error. This causes error to be a string containing <error Value>. Here is an example:

result, err := asyncResult.Get()
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        log.Println(err)
        return
    }

When I run this, I see <error Value> in the log where I print err. It is because in worker.go, on line 92, the second returned argument is converted to string, and the string representation of an error type is <error Value>.

If a task argument value of incorrect type is passed, the tasksignature.Get method get stuck waiting
for a result.

Example client:

    signatures.TaskArg{
            Type:  "int64",
            Value: "this-is-a-string",
    },

    asyncResult, err := server.SendTask(&task)
    if err != nil {
        fmt.Println(err)
    }

    result, err := asyncResult.Get()

The worker displays the following error:

2016/01/04 16:18:47 Failed processing task_488a2edb-433b-4fda-a4eb-d5a59568f3a3. Error = this-is-a-string is not int64
2016/01/04 16:18:47 Going to retry launching the worker. Error: Reflect task args: this-is-a-string is not int64

I think would be better to set the task state to failed, and return the error.

Hello @RichardKnop, I want to use my own broker (in memory broker) for testing purposes.
Could you add methods to Server for set up broker and backend?

    redisBroker.pool = redisBroker.newPool()
    defer redisBroker.pool.Close()

    _, err := redisBroker.pool.Get().Do("PING")
    if err != nil {
        redisBroker.retryFunc()
        return true, err // retry true
    }

I have a test case where i see how my program handles redis crashes. Once i identify the crash, i want to close the worker and create a new one since redis's ip might've changed.
So when redis crashes, i try to call Quit() in the hope of stopping the previous worker goroutine. The stopping channel isn't sampled, so the worker won't quit since it'll keep on retrying.

I tried to use send group task feature. However, it turns out to be quite slow. The scheduling frequency seems to be fixed. So I end up using send task one by one.

What is exact the benefit to use send group instead of sending task one by one?

There seems to be a race issue with chords when using Redis or Memcache backends.

I have observed an intermittent bug when chord callbacks would not get triggered. Probably caused by inconsistent state of TaskStateGroup.

Needs some investigation.

I suggest we keep the existing task UUID if included in a group.

Work with amqp not stable:
from time to time sender hangs on results.Get()
fix: ctrl+c and re-run again. some times from 3rd try.

Today I see worker crashed with this output:
/home/user/go/bin/src/github.com/RichardKnop/machinery/v1/brokers/amqp.go:161 +0x1c9

goroutine 13329 [select]:
github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(_Connection).heartbeater(0xc820076240, 0x2540be400, 0xc8200e6300)
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/connection.go:494 +0x505
created by github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(_Connection).openTune
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/connection.go:713 +0x805

goroutine 13330 [chan receive]:
github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.bufferDeliveries(0xc8200e65a0, 0xc8200e6540)
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/consumers.go:38 +0x7b
created by github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(*consumers).add
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/consumers.go:74 +0x10b

goroutine 13372 [runnable]:
github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(_Connection).heartbeater(0xc82391f320, 0x2540be400, 0xc8239754a0)
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/connection.go:494 +0x505
created by github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(_Connection).openTune
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/connection.go:713 +0x805

goroutine 13371 [runnable]:
net.runtime_pollWait(0x7fb9dba4ab98, 0x72, 0xc820010180)
/usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(_pollDesc).Wait(0xc823934610, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(_pollDesc).WaitRead(0xc823934610, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(_netFD).Read(0xc8239345b0, 0xc82393b000, 0x1000, 0x1000, 0x0, 0x7fb9dba45050, 0xc820010180)
/usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(_conn).Read(0xc820028628, 0xc82393b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
bufio.(_Reader).fill(0xc823975440)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(_Reader).Read(0xc823975440, 0xc82393d2d0, 0x7, 0x7, 0x1, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:207 +0x260
io.ReadAtLeast(0x7fb9dba4aef8, 0xc823975440, 0xc82393d2d0, 0x7, 0x7, 0x7, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:298 +0xe6
io.ReadFull(0x7fb9dba4aef8, 0xc823975440, 0xc82393d2d0, 0x7, 0x7, 0x1, 0x0, 0x0)
/usr/local/go/src/io/io.go:316 +0x62
github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(_reader).ReadFrame(0xc821d99ee0, 0x0, 0x0, 0x0, 0x0)
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/read.go:49 +0xbd
github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.(_Connection).reader(0xc82391f320, 0x7fb9dba4ae60, 0xc820028628)
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/connection.go:464 +0x172
created by github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp.Open
/home/user/go/bin/src/github.com/RichardKnop/machinery/Godeps/_workspace/src/github.com/streadway/amqp/connection.go:220 +0x3e3

amqp, server and workers are on the same server.

I'm trying to use machinery within a larger application. I was wondering if there was any way to launch machinery workers within the application, so that it can send and work on applications at the same time...

Or am I fundamentally misusing machinery?

Hey,

just a suggestion - use Sidekiq way of doing things to make this useful for a lot broader user base. Will help with adoption/migration for sure since a lot of people are using that.

I can document everything (how sidekiq works with redis) and also help with some implementation, although my Go skills leave a lot to be desired ;)

As far as I can see it's not possible at the moment to set the size of worker's pool? How do you plan to throttle incoming tasks to protect the resources?

Running it, happened this:

/code/go/src/github.com/RichardKnop/machinery$ go run examples/worker/worker.go
v1/amqp.go:11:2: cannot find package "github.com/streadway/amqp" in any of:
    /usr/lib/go/src/pkg/github.com/streadway/amqp (from $GOROOT)
    /home/vanhalt/code/go/src/github.com/streadway/amqp (from $GOPATH)
v1/config/config.go:7:2: cannot find package "gopkg.in/yaml.v2" in any of:
    /usr/lib/go/src/pkg/gopkg.in/yaml.v2 (from $GOROOT)
    /home/vanhalt/code/go/src/gopkg.in/yaml.v2 (from $GOPATH)

nats is a brokerless message queue

i woudl love to use this with it :)

current implementation for broker and backend are with factory that use connection prefix as backend selector and i think is not posible to extend the broker or backend? CMIIW how about
database/sql style backends and brokers?

I think it would be really awesome if Kafka can be added as a broker. I am currently investigating using kafka to send messages as part of an event-sourced system on mesos. Being able to use kafka would mean that existing kafka infrastructure can be used, rather than having to build a AMQP cluster on mesos.

Something simple where we can have this library use a logger that follows the standard interface would be good.

I'm looking to use logrus to hide a good chunk of the printf spam that I get from using this library.

@RichardKnop been working with Machinery for a few months now and it is really great. Thank you so much for your work on the project!

I am interested in the ability to query a broker for tasks that are currently enqueued. I've searched through the project and don't believe this is currently supported.

Unless I am mistaken, would you consider a PR that provides this functionality? I'm interested in this for testing purposes. IE I want to test some behavior that should publish a few tasks, then verify that they actually made it to the queue with the appropriate data.

For ex, I have the following implemented on a fork for the redis broker.

// PendingMessages returns a slice of task.Signatures currently enqueued.
func (redisBroker *RedisBroker) PendingMessages() ([]*signatures.TaskSignature, error) {
    conn, err := redisBroker.open()
    if err != nil {
        return nil, fmt.Errorf("Dial: %s", err)
    }
    defer conn.Close()

    bytes, err := conn.Do("LRANGE", redisBroker.config.DefaultQueue, 0, 10)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return nil, err
    }
    results, err := redis.ByteSlices(bytes, err)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return nil, err
    }

    var taskSignatures []*signatures.TaskSignature
    for _, result := range results {
        var taskSignature signatures.TaskSignature
        if err := json.Unmarshal(result, &taskSignature); err != nil {
            return nil, err
        }
        taskSignatures = append(taskSignatures, &taskSignature)
    }
    return taskSignatures, nil
}

I have additionally implement a function to purge a queue, which is also useful for testing.

// PurgeQueue
func (redisBroker *RedisBroker) PurgeQueue() error {
    conn, err := redisBroker.open()
    if err != nil {
        return nil
    }
    defer conn.Close()

    _, err = conn.Do("DEL", redisBroker.config.DefaultQueue)
    return err
}

Would love to know your thoughts on the above.

Publish Message: write tcp 192.168.56.1:14302->192.168.56.6:6379: wsasend: An existing connection was forcibly closed by the remote host.

I start a chord task which has more than 1500 subtask， one of each contains a 1K bytes message.

Is there a reason to open/close broker connection on every message publishing?
In a single goroutine with AMPQ connection reusing I have 110 msg publish/sec vs 55 msg/sec with current implementation.
Also with using of goroutines to publish messages I was able to get 34000 msg/sec on the same configuration. Looks like goroutines is safe when reusing AMPQ connection

Ability to delay the tasks for processing e.g run task in 5 Min.

Could be implemented by adding a Execute Time to the task signature, and by user a helper method e.g

func (t *TaskSignature) Delay(d time.Duration) 

When the worker gets the task, it holds onto it until the execution time and then runs it.

You want a store a time to execute and not just the delay as you want to guarantee that the task is run as close as possible to it, If the worker crashes the delay would start again otherwise.

file: ./worker.go
line: 139

if results[1].IsNil() 

this code on my machine will got an error "Error = runtime error: index out of range"

value of "results" is [<int Value>]

go version: go1.6.3 darwin/amd64

task:

func Add(a int, b int) int {
    return a + b
}

worker:

task := signatures.TaskSignature{
        Name: "add",
        Args: []signatures.TaskArg{
            {
                Type:  "int",
                Value: 1,
            },
            {
                Type:  "int",
                Value: 1,
            },
        },
    }
_, sendTaskErr := server.SendTask(&task)

I have a use-case where semi-sensitive data is passed around through a redis queue. This data should not be revealed in the logs, but https://github.com/RichardKnop/machinery/blob/master/v1/brokers/redis.go#L211 causes the data to appear.

Ideally it should be possible to suppress this output without suppressing all log output (currently it is possible to provide an alternative logger implementation but since the log line doesn't set any log level such as Debug, it can't be selectively filtered)

Using a custom broker seems to be the only possibility, but this is awkward since it requires copying the code of brokers/redis.go and factories.go to create an alternative redis implementation (an issue which seems to be related to #97).

Could a flag be added to denote data as sensitive, and therefore not to be logged anywhere? Alternatively, could that log line be changed to debug level (or other log lines be changed to warn level as appropriate) so that this could be filtered by the logger?

My task logic would send new tasks within a task body. In this case, this worker no longer receive no tasks until it finishes the current one. It can be workarounded by initiating multiple workers on one nodes but it seems strange.

To me, it's a waste of resource. But is it a valid scenario? Or should I avoid initiating tasks within the task body?

Is there an opportunity for a task to return a struct or map value in order to pass more information. For instance if I'm doing file processing tasks, I'd like to convey information like filepaths, sizes, stats, etc. Currently I'm just writing the data to a json file and sending back a string value of the config filepath as a result. Is it problematic for passing such a struct value to chained tasks?

Result backend to use for keeping task states and results. This setting is optional, you can run >Machinery without keeping track of task results.

But in fact when I try to leave ResultBackend empty there is a panic at

src/github.com/RichardKnop/machinery/v1/server.go:102 (0x2a4492)
    (*Server).SendTask: if err := server.backend.SetStatePending(signature); err != nil

The latest server.go changes the following error handling.

backend, err := BackendFactory(cnf)
if err != nil {
    return nil, err
}

It used to be:

// Backend is optional so we ignore the error
backend, _ := BackendFactory(cnf)

This change breaks my customised result backend. I still want to monitor the error returned by NewServer(). Can you revert it or provide an alternative to support customised result backend?

Print the stack when panic happens when processing task.

At the moment the stack is lost when a panic happens inside of a task. I do not know if its best to enforce people to put there own handling for panics or just update the global worker panic recover to have a stack trace, or both?

At the moment you can get into situations like this:

2016/08/18 07:56:45 Failed processing task_225ef62d-6c6b-42dd-9bce-d91d4c7edea9. Error = runtime error: invalid memory address or nil pointer dereference

Which makes it impossible to debug.

If its a good idea to put this in the global worker recover let me know and with make a pull request for it.

I have been going over tests recently and while the test coverage is decent, there is a lot of low hanging fruit to improve the tests. Make them more readable, reuse code.

I would like to use testify library instead of just Go's core testing module. I have been using testify in my other projects and it improves the tests readability and maintainability.

See: https://github.com/stretchr/testify

Chord callback is being called N times where N is number of tasks in the group.

This affects Redis and Memcache backends. AMQP backend is not affected.

Workaround until this is fixed is to make sure chord callback is idempotent.

You open a connection in AMQPBroker.Publish method, but you don't close it.
Why?

if i try the example with

func Add(args ...int64) (int64, error) {
    sum := int64(0)
    for _, arg := range args {
        sum += arg
    }
    return sum, nil
}

i get "Failed processing {54740152-8C08-4F6F-BA2A-579B092E0615}. Error = 1 is not int64", from worker.go

i think it is because the default behavior of encoding/json is when it finds a number in json and a interface in go it converts the number to float64

I was thinking of having a broker written entirely in Go. Its benefits would be:

• No external dependencies: you don't need Redis or AMPQ. It's all Go.
• You're not limited to marshalable data: I have to do some weird things because I can't pass a channel to my task handler function. If this was all in Go, I could pass any value I wanted.

Would it work with Machinery? Would there be a point in it? I was thinking that it would reduce a lot of boilerplate code in projects which want to do some asynchronous jobs such as sending emails, without depending on a messaging queue, or people like me, who want to pass in unmarshalable values to their functions.

Just FYI :(

Use vendoring for dependencies. See:

https://groups.google.com/forum/#!msg/golang-dev/nMWoEAG55v8/iJGgur7W_SEJ

Hi,
When I test with the example, I got this error:

Could not send task: Set State Pending: Queue Declare: Exception (406) Reason: "PRECONDITION_FAILED - invalid arg 'x-message-ttl' for queue 'task_5d2b652d-8428-4834-8c5e-bb4c6bc94d50' in vhost '/': {value_negative,-694967296}"
panic: Could not send task: Set State Pending: Queue Declare: Exception (406) Reason: "PRECONDITION_FAILED - invalid arg 'x-message-ttl' for queue 'task_5d2b652d-8428-4834-8c5e-bb4c6bc94d50' in vhost '/': {value_negative,-694967296}"

I am running RabbitMQ 3.6.4 on my mac and use the default exampleconfig.yml,

I also do search, the problem seems to be declare it again with a x-message-ttl, so i remove this line

results_expire_in: 3600000

then it works fine, is there other way to solve this problem?

thanks.

We've been evaluating different queuing alternatives for golang with a Redis backend. Machinery seemed to be promising, but we were worried about different priority queues usage.

We're building a system in which several workers should share a queuing system to pick and perform tasks. We want some tasks to have higher priority than others and workers to pick first from higher priority queues if they contain messages.

From the examples we reviewed, it seems machinery is only capable to work with a single default queue. Is there any way to implement the desired behaviour? In case not, is there any plan to include this at any time?

We had other systems written in Python and we switched from Celery to rq because of the priority management in the former.

Thanks!

I'd love to see some kind of scheduled task support, where you can submit a task for execution at a point of time in the future.

Awesome work on this project!

Did you have any plans to add the ability for a task to update its state with metadata? I'm currently using this project to import data and it would be beneficial to be able to determine the progress of the task. I wanted to gauge the interest in having this feature before implementing it. My initial thoughts would be to add a field into the TaskState struct which would hold any metadata on the task.

type TaskState struct {
  TaskUUID string
  State    string
  Result   *TaskResult
  Metadata map[interface{}]interface{}
  Error    string
}

There could then be a flag set in the task signature that would tell the worker to pass along a function that would be used by the task to set it's metadata.

type TaskSignature struct {
  UUID           string
  Name           string
  RoutingKey     string
  GroupUUID      string
  GroupTaskCount int
  Args           []TaskArg
  Immutable      bool
  HasMetadata    bool // tells worker to pass along a function to update metadata
  OnSuccess      []*TaskSignature
  OnError        []*TaskSignature
  ChordCallback  *TaskSignature
}

Having machinery run with tasks that have a long (+1 minute) I/O wait gets tedeously slow. It would be great to be able to increase the prefetch count.

For APMQ, this lives here:

Hi @RichardKnop , I'm moving from python to golang and I need a celery like architecture to handle asynchronous tasks for almost real time execution. Machinery looks a great fit for it, but I would like to get your honest review about whether it is ready to use in production and how much time it will take?

Seems like the Redis adaptor is using LPOP instead of BLPOP which leads to super busy polling.

So I have something like this:

    func init() {
    ...
        server, err := machinery.NewServer(&cnf)
        if err != nil {
            // some error handling
        }

        server.RegisterTasks(tasks)

        worker = server.NewWorker("machinery_worker")
    ...

    func main() {
        err := worker.Launch()
        if err != nil {
            // some error handling
        }
    }

which leads to this if the worker cannot connect to the broker on startup:

    2016/04/20 16:01:59 Launching a worker with the following settings:
    2016/04/20 16:01:59 - Broker: amqp://some_url
    2016/04/20 16:01:59 - ResultBackend: amqp://some_url
    2016/04/20 16:01:59 - Exchange: machinery_exchange
    2016/04/20 16:01:59 - ExchangeType: direct
    2016/04/20 16:01:59 - DefaultQueue: machinery_tasks
    2016/04/20 16:01:59 - BindingKey: machinery_task
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x20 pc=0x3495e9]

    goroutine 5 [running]:
    panic(0x4d3680, 0xc82000a0d0)
        /usr/local/go/src/runtime/panic.go:464 +0x3e6
    github.com/RichardKnop/machinery/vendor/github.com/streadway/amqp.(*Channel).Close(0x0, 0x0, 0x0)
        /Users/andy/checkouts/golang/src/github.com/RichardKnop/machinery/vendor/github.com/streadway/amqp/channel.go:402 +0x39
    github.com/RichardKnop/machinery/v1/brokers.(*AMQPBroker).StartConsuming(0xc8200165c0, 0x5d7220, 0x10, 0xc44828, 0xc8200bd500, 0x1, 0xc40028, 0xc8200e2050)
        /Users/andy/checkouts/golang/src/github.com/RichardKnop/machinery/v1/brokers/amqp.go:57 +0x156
    github.com/RichardKnop/machinery/v1.(*Worker).Launch.func1(0xc44740, 0xc8200165c0, 0xc8200bd500, 0xc820010540)
        /Users/andy/checkouts/golang/src/github.com/RichardKnop/machinery/v1/worker.go:38 +0xa2
    created by github.com/RichardKnop/machinery/v1.(*Worker).Launch
        /Users/andy/checkouts/golang/src/github.com/RichardKnop/machinery/v1/worker.go:47 +0x67d

I'm trying to figure out where the process dies. server.RegisterTasks(tasks) or in worker = server.NewWorker("machinery_worker") (I'm catching all other errors).

Although the output would suggest that the error actually occurs during worker.Launch(), which leads me to believe that there's a bug in the error handling (because err comes back as nil)?

Hi, is there way to limit count of concurrent tasks running on one worker server? For example: I know, thar server A is slow, but server B is fast and i want to server B perform more tasks than server A. I want to told machinery worker that it can perform only 10 goroutins on server A, and 20 on server B.

Or another method: on servers i have some metric(GPU memory usage) and i want to configure worker to accept new tasks only if that metric is low.

Any suggestions?
Thanks!

when I start server and there no workers with registered tasks which server sends, there many queries to amqp and near 100% cpu usage

I haven't nail down the root cause yet. But my worker seems to stopped working after it received a new message, and leave the message un-acknowledged. I tried purge the queue, and 3 were left un-acknowledged. Then I shutdown the worker, the 3 messages are changed back to ready.

I'm always use the latest code base and it has happened two times. Will try again to see if there is a pattern.

When a connection drops, worker will keep retrying to connect to the broker using Fibonacci sequence, so it will retry in 1, 1, 2, 3, 5, 8... seconds.

However, when worker successfully connects and starts consuming messages, the retry Fibonacci sequence should be reset so next time connection drops we start from 1 again.

/del

Running tests on Codeship fails. Investigate and try to find a source of the problem.

There have been intermitten issues with integration tests on Travis as well but that is less pressing.

It seems like RedisBroker.StopConsuming() doesn't properly shut down the Redis receiving goroutine.

This causes one of the integration tests to fail (I have commented it out for now).

Not a high priority issue as StopConsuming() methods are only used by integration tests for now.

Hello,

I'm working on an application which uses Machinery with a Redis backend, both embedded on a single desktop client. For a few reasons, we've concluded that using a unix domain socket is preferable to a network socket. Socket connections are already supported in redigo, so I'd like to propose extending that support to Machinery.

This may not fit perfectly with the current means of instantiating a backend/broker, but here are some thoughts:

• Add support for redis.DialOptions in RedisBackend or in the NewRedisBackend constructor. This would be the most open-ended and probably best way of customizing use of Redis, since it lets the user granularly set a password, database number, and connection options.
• The existing ParseRedisURL function wouldn't work well with file paths — currently it assumes that what comes after the final / is a DB number — so maybe a separate URL scheme would be needed. E.g.:

"redissocket:///path/to/file.sock"
"redissocket:///path/to/file.sock:/db" -- use ":" to separate final segment

In either case, the user should be able to supply redis.DialOptions to modify options even if the URL scheme can't.

I'd love to hear your thoughts on this proposal and am willing to work on a pull request if there is a design proposal that you find acceptable. Let me know what you think and how I can help!