d3sw / floop Goto Github PK

Travis should use the Makefile for consistency as well as testing the Makefile as well - https://github.com/d3sw/floop/blob/master/.travis.yml

Expected Behavior

A message is sent via the 'begin' handler, my testing shows the message sent is 'null.'
Floop should probably catch this and send a meaningful error.

Current Behavior

Currently it fails with message
"2018/03/26 17:55:12 [INFO] Registered handler: phase=completed handler=gnatsd
2018/03/26 17:55:12 [INFO] Registered handler: phase=progress handler=gnatsd
2018/03/26 17:55:13 [INFO] Registered handler: phase=failed handler=gnatsd
2018/03/26 17:55:13 [INFO] Registered handler: phase=begin handler=gnatsd
panic: interface conversion: interface {} is nil, not *types.ChildResult

goroutine 1 [running]:
github.com/d3sw/floop.(*phaseHandler).Handle(0xc420114a00, 0xc4200f3560, 0x94f7c0, 0x5, 0xc420130390)
/Users/Todd/go/src/github.com/d3sw/floop/handler.go:58 +0x19e
github.com/d3sw/floop.(*Lifecycle).Begin(0xc4200dbb60, 0xc4200f3530, 0x0, 0x0)
/Users/Todd/go/src/github.com/d3sw/floop/lifecycle.go:110 +0x183
github.com/d3sw/floop.(*Floop).Start(0xc4200f3500, 0xc420017bf0, 0xc4200f3500, 0x0)
/Users/Todd/go/src/github.com/d3sw/floop/floop.go:81 +0xb7
main.(*CLI).run(0xc42004c820, 0xc420017b90, 0xc420017b30, 0xc420017ad0)
/Users/Todd/go/src/github.com/d3sw/floop/cmd/cli.go:138 +0x162
main.(*CLI).Run(0xc42004c820, 0x7, 0xc42000e0b0, 0x9)
/Users/Todd/go/src/github.com/d3sw/floop/cmd/cli.go:111 +0x4b
main.main()
/Users/Todd/go/src/github.com/d3sw/floop/cmd/main.go:20 +0xfc"

Steps to Reproduce (for bugs)

1. Send a null through a handler using version 1.3

Context

Your Environment

Alpine docker container

• Product Version: openjdk:8u121-jdk-alpine

Expected Behavior

Progress gets reported when underlying process writes progress to stderror.

Current Behavior

It doesn't show progress with programs that write progress to stderror

Possible Solution

A possible solution is to add a flag that will allow progress if it is written to stderror.

Steps to Reproduce (for bugs)

1. Configure floop to show progress.
2. Run a programs that logs to stderror with floop.
3. Check progress messages.

Context

Trying to collect progress from Apple iTunes Transporter.

Your Environment

• Product Version: v0.1.0
• OS and OS Version: CentOS release 6.9 (2.6.32-696.6.3.el6.x86_64)

-ldflags are required for dist builds. See other Deluxe open-source projects for reference.

Your issue may already be reported!
Please search on the issue track before creating one.

Expected Behavior

README or other docs should include an example where the configuration has required parameters and the command-line call passes them in to floop.

Current Behavior

There are currently no examples showing how to use parameters.

Possible Solution

Update the README, fool!

Your Environment

Sunny Southern California.

All NATS messages needs to be persisted so we need to update all NATS messages to NATS Streaming format.

This is a dependency for https://jira.d3nw.com/browse/ONETRNS-386

Expected Behavior

The configuration file should have a setting to enable HTTP requests to be retried upon failure. The configuration should also have a setting for the number of retries to attempt and a setting for the time interval to wait between attempts.

With this configuration option enabled Floop should retry the failed HTTP requests. If service discovery is enabled then Floop needs to do the service discovery for each attempt in order to avoid hitting service instances that are in a bad state.

When floop receives a SIGINT or SIGTERM it should

• propagate the signal to the child process
• appropriately handle cleanup, then shutdown.

Issue #12 is caused due to the above not being handled

I have a file path that I can't seem to pass in as a command line argument.

In order to pass the file path from the command line I add "${Meta.FILE_ID}" to the arguments section in config.yml

Here's a snippet of my config.yml
meta:

• FILE_ID

# Command to be run as child process
command: /app/checksumd
# Arguments passed to the command
args: [
"-file", "${Meta.FILE_ID}", "-all"
]

Here's my output:
C:\Projects\Go\src\github.com\d3sw\checksum>docker run --net="host" --rm -v D:\storage\staging\Vault:/data checksum FILE_ID=/data/movie.ts
2017/10/28 00:07:45 [INFO] Registered handler: phase=progress handler=gnatsd
2017/10/28 00:07:45 [INFO] Registered handler: phase=completed handler=gnatsd
2017/10/28 00:07:45 [INFO] Registered handler: phase=failed handler=gnatsd
2017/10/28 00:07:45 [INFO] (child) spawning: /app/checksumd -file ${Meta.FILE_ID} -all
2017/10/28 00:07:45 [ERROR] phase=progress handler=gnatsd transform: no matching data
[gnatsd] phase=failed topic=deluxe.checksum.status &{Code:2 Stdout:[67 97 110 110 111 116 32 111 112 101 110 32 102 105 108 101 58 32 36 123 77 101 116 97 46 70 73 76 69 95 73 68 125] Stderr:[112 97 110 105 99 58 32 105 110 118 97 108 105 100 32 97 114 103 117 109 101 110 116 10 10 103 111 114 111 117 116 105 110 101 32 49 32 91 114 117 110 110 105 110 103 93 58 10 103 105 116 104 117 98 46 99 111 109 47 100 51 115 119 47 99 104 101 99 107 115 117 109 47 99 104 101 99 107 115 117 109 46 40 42 67 104 101 99 107 115 117 109 41 46 67 97 108 99 117 108 97 116 101 72 97 115 104 101 115 40 48 120 99 52 50 48 48 50 102 102 49 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 48 120 48 44 32 46 46 46 41 10 9 67 58 47 80 114 111 106 101 99 116 115 47 71 111 47 115 114 99 47 103 105 116 104 117 98 46 99 111 109 47 100 51 115 119 47 99 104 101 99 107 115 117 109 47 99 104 101 99 107 115 117 109 47 99 104 101 99 107 115 117 109 46 103 111 58 57 55 32 43 48 120 53 98 53 10 109 97 105 110 46 109 97 105 110 40 41 10 9 67 58 47 80 114 111 106 101 99 116 115 47 71 111 47 115 114 99 47 103 105 116 104 117 98 46 99 111 109 47 100 51 115 119 47 99 104 101 99 107 115 117 109 47 109 97 105 110 46 103 111 58 52 52 32 43 48 120 49 54 48]}

If I hard code the file path as in the argument section it works:

meta:

• FILE_ID

# Command to be run as child process
command: /app/checksumd
# Arguments passed to the command
args: [
"-file", "/data/movie.mpg", "-all"
]

C:\Projects\Go\src\github.com\d3sw\checksum>docker run --net="host" --rm -v D:\storage\staging\Vault:/data checksum FILE_ID=/data/movie.ts
2017/10/28 00:15:11 [INFO] Registered handler: phase=progress handler=gnatsd
2017/10/28 00:15:11 [INFO] Registered handler: phase=completed handler=gnatsd
2017/10/28 00:15:11 [INFO] Registered handler: phase=failed handler=gnatsd
2017/10/28 00:15:11 [INFO] (child) spawning: /app/checksumd -file /data/movie.ts -all
[gnatsd] phase=progress topic=deluxe.checksum.progress map[Pfff:110100000020000100FFFF0F0001:F791C37D3970A63B4948897C4416D0CA Md5:12babc6149da3b5ba7afafe6120e8c44 Sha1:a667c61ef6564fad77418b14bcd8e7ea01368978 Sha256:41a5da6bee974bf6793d61689b2c051513efec4343cb441e75298188f9458605 Sha512:6d52fb8da9c4bbd30688c7cc7d1f33d771e9362a36846e14e6c692c0626c4fd67c7667fcaeaab7bc8d80494a8def1720df3d290611c3cf9f872ae25129e68694]
[gnatsd] phase=completed topic=deluxe.checksum.status map[Sha512:6d52fb8da9c4bbd30688c7cc7d1f33d771e9362a36846e14e6c692c0626c4fd67c7667fcaeaab7bc8d80494a8def1720df3d290611c3cf9f872ae25129e68694 Pfff:110100000020000100FFFF0F0001:F791C37D3970A63B4948897C4416D0CA Md5:12babc6149da3b5ba7afafe6120e8c44 Sha1:a667c61ef6564fad77418b14bcd8e7ea01368978 Sha256:41a5da6bee974bf6793d61689b2c051513efec4343cb441e75298188f9458605]

My config.yml handlers:

handlers:
completed:

• type: gnatsd
  uri: "nats://events.service.owf-dev:4222"
  options:
  topic: deluxe.checksum.checksums.status
  transform: ["json"]
  body: ${Data|json}
  failed:
• type: gnatsd
  uri: "nats://events.service.owf-dev:4222"
  options:
  topic: deluxe.checksum.checksums.status
  transform: ["json"]
  body: ${Stderr|json}"

Nats message returned:
"ewogICJpZCI6ICJmZWRlNjVmOC0xNzgxLTQxMjItYjA3OS0wM2Q1OTMyNTBiNjAiLAogICJwYXJlbnRpZCI6IG51bGwsCiAgInNvdXJjZSI6ICJNZWRpYUluZm8iLAogICJzdWJqZWN0IjogImRlbHV4ZS5tZWRpYWluZm8uaW5mby5zdGF0dXMiLAogICJuYW1lIjogImRlbHV4ZS5tZWRpYWluZm8uaW5mby5zdGF0dXMiLAogICJ2ZXJzaW9uIjogIjAuMC4xIiwKICAidGltZXN0YW1wIjogIjIwMTctMTEtMTVUMDE6MzE6NTJaIiwKICAidXJucyI6IFsKICAgICJkZWx1eGU6Y29uZHVjdG9yOndvcmtmbG93Onh5ejMyMSIKICBdLAogICJ0YWdzIjogbnVsbCwKICAicHJvZ3Jlc3MiOiBudWxsLAogICJzdGF0dXMiOiB7CiAgICAiTmFtZSI6ICJmYWlsZWQiLAogICAgIlJlYXNvbiI6ICJEZWx1eGUuRG90TmV0Q29yZS5NdmMuVXRpbGl0aWVzLkdsb2JhbEV4Y2VwdGlvbkZpbHRlci5BcGlFeGNlcHRpb246IFBhdGggZG9lcyBub3QgZXhpc3Q6ICAvZGF0YS9tb3ZpZS50czFcbiAgIGF0IERlbHV4ZS5Eb3ROZXRDb3JlLk1lZGlhSW5mby5Nb2RlbHMuTWVkaWFJbmZvV3JhcHBlci5HZXRNZWRpYUluZm8oU3RyaW5nIGFic29sdXRlUGF0aCwgQm9vbGVhbiBmdWxsUGFyc2UpXG4gICBhdCBNZWRpYUluZm8uV2ViQXBpLkNMSS5CYXRjaENvbW1hbmRQcm9jZXNzb3IuUnVuKCkgaW4gL21lZGlhaW5mby9NZWRpYUluZm8vTWVkaWFJbmZvLldlYkFwaS9DTEkvQmF0Y2hDb21tYW5kUHJvY2Vzc29yLmNzOmxpbmUgMzAiCiAgfSwKICAiZGF0YSI6IHsKICAgICJtZWRpYWluZm8iOiBudWxsLAogICAgIm1lZGlhaW5mb2NvbXBsZXRlIjogbnVsbCwKICAgICJ2ZXJzaW9uIjogIjAuMC4xIiwKICAgICJhcHB2ZXJzaW9uIjogbnVsbCwKICAgICJqb2JpZCI6IG51bGwKICB9Cn0="

Expected Behavior

Each handler for a given lifecycle phase should be executed concurrently.

Current Behavior

Each handler is executed sequentially

Possible Solution

Run each handler in it's own go-routine and wait for all to complete.

Expected Behavior

When the Floop process is killed (by Nomad delete job API call) then the child processes should be killed too.

Current Behavior

When the Floop process is killed (by Nomad delete job API call) then the child processes continues to execute even if the Floop process is no longer running.

Possible Solution

Check child process kill signal?

Steps to Reproduce (for bugs)

1. Start a long running process with Floop using nomad as a scheduler.
2. Once processing started send a Delete job command to nomad.
3. The child process will continue to run even if the Floop process is no longer running.

Context

Trying to stop an Apple iTunes Transporter transfer using the Nomad API.

Your Environment

• Product Version: v0.1.0
• OS and OS Version: CentOS release 6.9 (2.6.32-696.6.3.el6.x86_64)

For most of the transcoder job, we would like to sent the NATs message when the job is cancelled or interrupted(nomad -> floop -> transcoder_proxy -> transcoder). This will allow us to manage/send the correct status message to the NATs when job is cancelled. Currently status for complete, progress and failed can be sent - but would also need the handler for "Cancel".

deluxe.checksum.status
{
"Details": {
"Code": 2,
"Stdout": "RXJyb3I9b3BlbiBEOlxzdG9yYWdlXHN0YWdpbmdcVmF1bHRcbW92aWUubXBnMTogVGhlIHN5c3RlbSBjYW5ub3QgZmluZCB0aGUgZmlsZSBzcGVjaWZpZWQu",
"Stderr": null
},
"status": "FAILED"
}

Expected Behavior

Floop should be able to use service discovery. The configuration should have a flag that indicates if service discovery is needed for the domain to where floop needs to send the state of a job.

Current Behavior

Floop doesn't support service discovery.

Possible Solution

Use https://github.com/euforia/go-srv-resolver to add this functionality to floop.

Context

When a service to report to is deployed using Docker then it URI will change over time. By adding service discovery to Floop, it will be able to report progress even if the uri of the service changes.

Your Environment

• Product Version: v0.1.0
• OS and OS Version: CentOS release 6.9 (2.6.32-696.6.3.el6.x86_64)

A new config option should be added to the progress handler config

stderr: true

This would enable reading from stderr as well.

Core logic is here : https://github.com/d3sw/floop/blob/master/floop.go#L34-L35