kraken-hpc / kraken Goto Github PK

Kraken is a distributed state engine framework for scalable automation and orchestration tools.

License: BSD 3-Clause "New" or "Revised" License

Go 99.73% Shell 0.27%

hpc state-engine kraken boot state-synchronization eventually-consistent automation distributed-systems microservices provisioning distributed-automation

kraken's Introduction

Kraken

Reorganization notice

The Kraken project has been subdivided into different repos. This repo now only contains core components for the kraken framework; most modules and extensions have been moved elsewhere.

If you're trying to provision clusters, you probably want kraken-layercake
If you're looking for thermal control or Raspberry Pi modules/extensions, try kraken-legacy
If you're looking for ansible roles for kraken, you probably want kraken-layercake-ansible

What is Kraken?

Kraken is a distributed state engine framework for building tools that can maintain and manipulate state across a large set of computers. It was designed to provide full-lifecycle maintenance of HPC compute clusters, from cold boot to ongoing system state maintenance and automation (see: kraken-layercake for this implementation).

Kraken was designed for HPC use-cases but may be useful anywhere distributed automation is needed. Kraken is designed to be highly modular and should be able to adapt to many situations.

Kraken is modular

Kraken on its own is only a framework. When combined with modules, kraken can do things. Kraken modules can discover real state values in a system an communicate them. Kraken modules can declare that they know how to "mutate" the system to different states. Kraken modules can also do things like run tiny services the system needs.

Kraken state is extensible

Kraken starts with a very simple state-definition per node. Through extensions kraken can define new kinds of state information. You can loosely think of extensions as something like database schemas.

What do you mean by "state engine", and how does it work?

Kraken maintains a copy of the desired state (called "Configuration" state, abr "Cfg"). It also is able--through specialized modules--to discover bits of current real state ("Discoverable" state, abr. "Dsc"). Kraken modules provide a list of state mutations that they can perform, e.g. for PhysState: POWER_OFF to PhysState: POWER_ON. These mutations are used to generate a directed graph. At any time, if a difference is detected between Configuration (intended) state and Discoverable (actual) state, Kraken computes a path of mutations to converge on the Configuration state.

What do you mean by "distributed state engine", and how does that work?

Kraken distributes the state across potentially thousands of individual physical nodes. It maintains synchronization through a one-way state-update protocol that is reminiscent of routing protocols like OSPF. State synchronization in Kraken follows the "eventual consistency" model; we never guarantee that the entire distributed state is consistent, but can provide conditional guaranties that it will converge to consistency.

How do I learn more?

Kraken is in very active development. As part of the development efforts of Kraken, we will be updating the repository with more and more documentation, ranging from implementation guides to application architecture and module API guides. There are also a number of talks, papers, and presentations out there about Kraken and its related projects.

Notes on this version of Kraken

Kraken is still a fledgling sea-monster, but it has show itself capable of some pretty powerful things. The kraken-layercake project is our reference project for kraken capabilities. It can boot and maintain large scale compute clusters, providing unique capalities like:

Stateful rolling updates of images in microseconds
Self-healing capabilities at all layers of the stack
Active feedback and monitoring of system state through tools like kraken-dashboard and krakenctl

Check back soon for more documentation, utilities, and demonstrations.

Generating a Kraken-based app

A Kraken-based app consists of three core componenets (and maybe more):

An application entry point, i.e. where a main() function lives.
A set of extensions that specify extra state variables for nodes.
A set of modules that define mutations and discover states.

The kraken command can be used to generate source code for these components.

First, get kraken with:

go get -u github.com/kraken-hpc/kraken

You'll also want to make sure $GOPATH/bin is in your executuion path, e.g. export PATH=$PATH:$GOPATH/bin .

Generating an entry-point

You need an app definition to create an app. This contains metadata about the app (like it's name and version) as well as references for what extensions and modules to include.

Here's an example for an app called tester:

name: tester
version: "v0.1.1"
extensions:
  - "github.com/kraken-hpc/kraken/extensions/ipv4"
modules:
  - "github.com/kraken-hpc/kraken/modules/restapi"
  - "github.com/kraken-hpc/kraken/modules/websocket"

The canonical structure is to place this file in the directory where the entrypoint code should live, and name it kraken.yaml, but the naming is optional. Assuming this convention:

$ mkdir tester
$ cd tester
<create kraken.yaml>
$ kraken app generate
INFO[0000] app "tester" generated at "."

Now you can build your application:

$ go build .
$ ./tester -version
tester version: v0.1.1
this kraken is built with extensions: 
        type.googleapis.com/IPv4.IPv4OverEthernet
this kraken is built with modules: 
        github.com/kraken-hpc/kraken/modules/restapi
        github.com/kraken-hpc/kraken/modules/websocket

Generating a module

You also need a definition to create a module using the kraken command. The default name for this definition is module.yaml in the module directory. Canonically, we use <project>/modules/<module> for module directories.

Module defintions are a more complicated than app definitions. They need to define all of the mutations and discoveries that a module can do. Here's an annotated example:

---
# The package_url is the Go-style path to the module.  This must be a full url.
package_url: "github.com/kraken-hpc/kraken/modules/test"
# If with_polling is true, the module will be generated with a polling loop.
# The timer for the polling loop uses a config, so with_config is implied.
with_polling: true
# If with_config is specified, a stub for a protobuf config will be generated.
with_config: true
# This list declares any URLs we descover.
# The state of our own service and any URLs used in `mutates` sections of mutations below are automatically added.
# In this case, we're letting Kraken know that we discover things about `/RunState`, even though it's not part
# of our mutations.
discoveries:
  - "/RunState"
# This section declares mutations.  The key can be anything, but it must be unique. Ideally, it should be something descriptive.
# In this example we declare two mutations, one discovers /PhysState when it's unknown, the other mutates power from OFF to ON.
mutations:
  "Discover":
    mutates:
      "/PhysState":
        from: "PHYS_UNKNOWN"
        to: "POWER_OFF"
    requires:
      "/Platoform": "test"
    timeout: "10s"
    fail_to:
      url: "/PhysState"
      value: "PHYS_ERROR"
  "PowerON":
    mutates:
      "/PhysState":
        from: "POWER_OFF"
        to: "POWER_ON"
    requires:
      "/Platform": "test"
    timeout: "10s"
    fail_to:
      url: "/PhysState"
      value: "PHYS_ERROR"

Once we have the definition in place, we can generate the module with:

$ mkdir -p modules/test
$ cd modules/test
<create module.yaml>
$ kraken module generate 
INFO[0000] module "test" generated at "."

If we selected with_config: true, we will need to generate the protobuf code from the provided proto file. You can add some variables to test.config.proto first, then:

$ cd modules/test
$ go generate

This will create test.config.pb.go (note: you need protoc and gogo-proto installed).

At this point, the module can be built and run, though it won't really do anything. It will add to the generated graph, so this is a good point to make sure the graph output of an app that links this module is sane.

Finally, you'll want to edit test.go. Unlike test.mod.go, which shouldn't be edited by hand, test.go is a stub to get you started and you'll need to edit it. You'll need to add real function hanlders for your mutations in func Init(). If you made a polling loop you probably want to put some stuff in func Poll(). In general, look for comments starting with // TODO: for areas where you might want to alter things.

If you change your module definition, you can update your module without overwriting test.go.

$ kraken module update

This will only update test.mod.go. Note: you may need to make manual changes to make test.go match, e.g. if you changed your list of mutations.

Generating an extension

Extensions are the least complicated to generate. The definition file for an extension looks like:

---
package_url: github.com/kraken-hpc/kraken/test
name: TestMessage
custom_types:
  - "MySpecialType"

This will generate an extension that will be referenced as Test.TestMessage. Note that we support generating multiple extensions in the same proto package using multiple definition files. E.g., you could also have Test.AnotherMessage defined in another file.

The procedure is similar to the others. The default file name for extensions is extension.yaml:

$ mkdir -p extensions/test
$ cd extensions/test
<create extension.yaml>
$ kraken extension generate 
INFO[0000] extension "Test.TestMessage" generated at "."

I want to get involved...

Excellent! It's our intention to make Kraken a community developed project. To get started, you can:

contact us; Kraken has a Slack instance. You can get an invite here: slack.kraken-hpc.io
take a look at any posted issues;
post new issues;
create pull requests.

Enjoy!

kraken's People

Contributors

Stargazers

Watchers

Forkers

jlowellwofford kpelzel paulutk peltz-lanl relisnyder cluening caskie bensallen synackd ghazanfarttu zeta1999 johan-kok macdaliot mej linecode

kraken's Issues

Should we migrate CircleCI to Github Actions?

Need some investigation into Actions to see if this would do what we need.

BulkUpdateByType returns an invalid reflect value

BulkUpdateByType creates a NewStateChangeEvent, but passes in an empty reflect value. This will result in the debug message containing an invalid reflect value.

https://github.com/hpc/kraken/blob/9dacb64ab6b0eaee8d8971e3fbcbd23f70a6c426/core/StateDifferenceEngine.go#L530

Restapi: Implement PATCH for simple node modifications

It would be nice to not have to pass the entire node state to the API just to change one value. Allow the PATCH HTTP request to modify only select values.

Mutations & discoverables should be associated with a ServiceInstance, not a Module

Currently Mutations and Discoverables are registered to modules. This prevents having more than one ServiceInstance of the same module, and is just logically wrong.

Consider Migration to Go Modules

It would be useful to migrate to Go Modules, eg:

go mod init
go mod vendor

The primary benefit is no longer having to make use of GOPATH.

Tree sync

In order to fully support hierarchical systems, we need a "tree sync" mechanism to synchronize bulk child state with parents of the intermediate node.

SI refactor breaks build

2019/06/15 08:10:24 Run: go build -o main includes.go main.go pprof.go

command-line-arguments

./main.go:129:33: cannot use k (type *core.Kraken) as type lib.ServiceInstance in argument to core.Registry.RegisterMutations:
*core.Kraken does not implement lib.ServiceInstance (missing Cmd method)
./main.go:130:36: cannot use k (type *core.Kraken) as type lib.ServiceInstance in argument to core.Registry.RegisterDiscoverable:
*core.Kraken does not implement lib.ServiceInstance (missing Cmd method)

StateDifferenceEngine announces a state change even if we discover the already existing state

The StateDifferenceEngine should not fire a state change event if we discover something we already knew.

Nodes gets stuck in boot cycle if kraken dies

Booted the vbox example and sshed into the node; killed kraken service and the node went into an infinite booting loop. The loop was stopped by manually turning off node through the dashboard.

layer1 Dockerfile assumes ssh keys and munge key exist

utils/layer1/centos-supervisord/Dockerfile changes permissions on some files that aren't included in the distribution:

...
    && chmod 0644 /etc/ssh/*.pub /etc/ssh/moduli \
    && chmod 0640 /etc/ssh/ssh_*key \
    && chgrp ssh_keys /etc/ssh/ssh_*key \
...
   && chmod 0600 /etc/munge/munge.key \
...

The README notes that you can pull those files out of an already-built image and reuse them, but bootstrapping fails if they are not already in the repo.

A patch to fix this problem will be coming shortly.

SME will re-emit a mutation if it gets duplicate or irrelevant discoveries

This is happening with INITtoSYNC when we get both PXE -> INIT and RunState -> INIT.

19:00:04.401:kraken:DEBUG:EventDispatchEngine:dispatching event: STATE_MUTATION a0f54069-d31b-cb51-81e6-3f5c1468e5df (MUTATE) a0f54069-d31b-cb51-81e6-3f5c1468e5df : sse -> INITtoSYNC
19:00:04.404:kraken:DEBUG:EventDispatchEngine:dispatching event: STATE_MUTATION a0f54069-d31b-cb51-81e6-3f5c1468e5df (MUTATE) a0f54069-d31b-cb51-81e6-3f5c1468e5df : sse -> INITtoSYNC
19:00:41.562:kraken:DEBUG:EventDispatchEngine:dispatching event: DISCOVERY a0f54069-d31b-cb51-81e6-3f5c1468e5df:/RunState (sse) a0f54069-d31b-cb51-81e6-3f5c1468e5df:/RunState == SYNC
19:00:41.562:kraken:DEBUG:StateSyncEngine:successful phone home for: a0f54069-d31b-cb51-81e6-3f5c1468e5df

Restapi: Allow puts to upgrade to posts

...i.e. if a node doesn't exist, create it. This allows for a more idempotent interface.

Consider migrating module launching to go-fork

This might resolve issues mentioned in #185

Find better solution than sprintf in lib.Node diff function

#140 replaces lib.urlpush with sprintf to avoid a beginning slash. A beginning slash will cause issues with the SME because it filters out any STATE_CHANGE events that contain extensions with a leading slash. This solution is not ideal, but was passed in due to a time crunch.

related to PR: #115

These problems occur in lib.Node.Diff()

	// handle extensions
	for _, u := range eright {
		nodeExt, ok := m.exts[u]
		if !ok {
			r = append(r, lib.URLPush(prefix, u))
			r = append(r, fmt.Sprintf("%s%s", prefix, u))
			continue
		}
		d, _ := lib.MessageDiff(n.exts[u], nodeExt, lib.URLPush(prefix, u))
		d, _ := lib.MessageDiff(n.exts[u], nodeExt, fmt.Sprintf("%s%s", prefix, u))
		r = append(r, d...)
		for i := range eleft {
			if eleft[i] == u {
@@ -399,9 +407,11 @@ func (n *Node) Diff(node lib.Node, prefix string) (r []string, e error) {
		}
	}
	for _, u := range eleft {
		r = append(r, lib.URLPush(prefix, u))
		r = append(r, fmt.Sprintf("%s%s", prefix, u))

CentOS-Base.repo uses unreachable repos

In utils/layer1/centos-supervisord/repos/CentOS-Base.repo the listed repos are unreachable for me. A simple change back to the original list of repos let the docker build complete.

I'll submit a PR from my fork in a minute.

Send SSDP Packet When Booting Raspberry PI

The raspberry pis 3Bs don't like to send a dhcp discover until they are hit with an SSDP packet.

Add functionality to the PiPXE module so that it will send an SSDP packet on the network when it gets a NONEtoWAIT mutation request.

Erroneous error messages if we discover that we're already in the desired state

If we want to be in state X and a change in discoverable state finds us already there, we get erroneous error messages about finding a path in the mutation graph:

15:41:29.620:kraken:DEBUG:EventDispatchEngine:dispatching event: DISCOVERY 123e4567-e89d-12d3-a456-425455440096:/PhysState (github.com/hpc/kraken/modules/capmc) 123e4567-e89d-12d3-a456-425455440096:/PhysState == POWER_ON
15:41:29.620:kraken:DEBUG:StateDifferenceEngine:discovered 123e4567-e89d-12d3-a456-425455440096:/PhysState is POWER_ON
15:41:29.620:kraken:DEBUG:EventDispatchEngine:dispatching event: STATE_CHANGE 123e4567-e89d-12d3-a456-425455440096:/PhysState (UPDATE) 123e4567-e89d-12d3-a456-425455440096:/PhysState = <proto.Node_PhysState Value>
15:41:29.620:kraken:ERROR:StateMutationEngine:path not found: you can't get there from here

Where, in the above example we want POWER_ON and also discover POWER_ON.

Instead, the mutation check should notice that we're already where we want to be and do nothing.

Re-organize documentation

Currently, we have README.md files lurking around all over the place.

It would probably be much better to make a docs/ folder, and placed appropriately named docs there.

SSE throws error when it shouldn't

If we, e.g. discover power is off, we drop out of SYNC. The SSE doesn't necessarily catch this right away, and may lose sync and throw an error it shouldn't. This can overly complicate the recovery process by throwing an additional unexpected state change into the chain.

Dynamically detect kraken's import path

When compiling templates for u-root, import paths in the compiled templates are changed from using packages in the main kraken source tree (i.e. github.com/hpc/kraken/... to using the similar packages that are included with the generated sources (i.e. github.com/hpc/kraken/build/u-root/....

This is currently hard-coded, and this should ideally be detected dynamically.

PhoneHome should use TLS

...otherwise secret keys are passed in the clear.

Should lib/util merge/diff functions use protoreflect?

Seems like the new canonical thing to do... that said, what we have seems to work.

Add option to create configured kraken source tree (for u-root)

In order to be able to have kraken be a u-root command (for future firmware aspirations), it would be useful to add an option to kraken-build to generate a u-root-embeddable source tree.

Off-by-one in SME

When handling unexpected discoveries, there appears to be an off-by-one error in the SME that eventually tries to advance a chain that has no more mutations. Results in:

08:58:09.303:kraken:DEBUG:StateMutationEngine:123e4567-e89d-12d3-a456-425455440096 found a new path
panic: runtime error: index out of range

goroutine 24 [running]:
github.com/hpc/kraken/core.(*StateMutationEngine).advanceMutation(0xc0001c6000, 0xc0000c73e0, 0x24, 0xc000394b80)
	/home/vagrant/go/src/github.com/hpc/kraken/core/StateMutationEngine.go:730 +0x61a
github.com/hpc/kraken/core.(*StateMutationEngine).handleUnexpected(0xc0001c6000, 0xc0000c73e0, 0x24, 0xc0000c7405, 0x9, 0x9ee0e0, 0xc00041e6e8, 0x185)
	/home/vagrant/go/src/github.com/hpc/kraken/core/StateMutationEngine.go:818 +0xded
github.com/hpc/kraken/core.(*StateMutationEngine).updateMutation(0xc0001c6000, 0xc0000c73e0, 0x24, 0xc0000c7405, 0x9, 0xab7c60, 0xc00038f260, 0x99)
	/home/vagrant/go/src/github.com/hpc/kraken/core/StateMutationEngine.go:878 +0x1175
github.com/hpc/kraken/core.(*StateMutationEngine).handleEvent(0xc0001c6000, 0xb771a0, 0xc0001dca50)
	/home/vagrant/go/src/github.com/hpc/kraken/core/StateMutationEngine.go:971 +0x336
github.com/hpc/kraken/core.(*StateMutationEngine).Run(0xc0001c6000)
	/home/vagrant/go/src/github.com/hpc/kraken/core/StateMutationEngine.go:286 +0x6c8
created by github.com/hpc/kraken/core.(*Kraken).Run
	/home/vagrant/go/src/github.com/hpc/kraken/core/Kraken.go:198 +0x10a
got stream read error on mutation stream: rpc error: code = Unavailable desc = transport is closing
got stream read error on mutation stream: rpc error: code = Unavailable desc = transport is closing
got stream send error on logger stream: EOF
got stream send error on logger stream: EOF

Add sdnotify support

When running under systemd, it would be nice if kraken told systemd once it was initialized...

concurrent map iteration panic with DEBUG logging and large number of nodes

When in log >= 7 and a large number of nodes acting at once, can arrive at the following concurrency fatal error.

fatal error: concurrent map iteration and map write

goroutine 22 [running]:
runtime.throw(0x15ff1a0, 0x26)
/Users/lowell/dev/goroot/src/runtime/panic.go:608 +0x72 fp=0xc000309160 sp=0xc000309130 pc=0x102c112
runtime.mapiternext(0xc0005a4540)
/Users/lowell/dev/goroot/src/runtime/map.go:790 +0x525 fp=0xc0003091e8 sp=0xc000309160 pc=0x1010495
reflect.mapiternext(0xc0005a4540)
/Users/lowell/dev/goroot/src/runtime/map.go:1277 +0x2b fp=0xc000309200 sp=0xc0003091e8 pc=0x101133b
reflect.Value.MapKeys(0x152e880, 0xc0003a1590, 0x1b5, 0xb4, 0xc000000076, 0x3)
/Users/lowell/dev/goroot/src/reflect/value.go:1134 +0x12d fp=0xc000309298 sp=0xc000309200 pc=0x10afe5d
fmt.(*pp).printValue(0xc000312e40, 0x152e880, 0xc0003a1590, 0x1b5, 0x76, 0x2)
/Users/lowell/dev/goroot/src/fmt/print.go:746 +0xd66 fp=0xc000309478 sp=0xc000309298 pc=0x10c5656
fmt.(*pp).printValue(0xc000312e40, 0x158e5c0, 0xc0003a1580, 0x199, 0xc000000076, 0x1)
/Users/lowell/dev/goroot/src/fmt/print.go:783 +0x1c92 fp=0xc000309658 sp=0xc000309478 pc=0x10c6582
fmt.(*pp).printValue(0xc000312e40, 0x15d4d60, 0xc0003a1580, 0x16, 0xc000000076, 0x0)
/Users/lowell/dev/goroot/src/fmt/print.go:853 +0x1ad5 fp=0xc000309838 sp=0xc000309658 pc=0x10c63c5
fmt.(*pp).printArg(0xc000312e40, 0x15e88e0, 0xc0002f0960, 0x76)
/Users/lowell/dev/goroot/src/fmt/print.go:683 +0x3d4 fp=0xc0003098d0 sp=0xc000309838 pc=0x10c42a4
fmt.(*pp).doPrintf(0xc000312e40, 0x15e999d, 0x2, 0xc000309a48, 0x1, 0x1)
/Users/lowell/dev/goroot/src/fmt/print.go:1003 +0x166 fp=0xc0003099b8 sp=0xc0003098d0 pc=0x10c7d26
fmt.Sprintf(0x15e999d, 0x2, 0xc000309a48, 0x1, 0x1, 0xc00019a840, 0x9ef5ceb0)
/Users/lowell/dev/goroot/src/fmt/print.go:203 +0x66 fp=0xc000309a10 sp=0xc0003099b8 pc=0x10c0ab6
github.com/hpc/kraken/lib.ValueToString(0x15d4d60, 0xc0003a1580, 0x16, 0xc0000cf370, 0x1547020)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/lib/util.go:290 +0xec fp=0xc000309ab0 sp=0xc000309a10 pc=0x13ff26c
github.com/hpc/kraken/core.(*StateChangeEvent).String(0xc00019a840, 0x16102d0, 0xc0003a6000)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateDifferenceEngine.go:50 +0x97 fp=0xc000309b48 sp=0xc000309ab0 pc=0x143e537
fmt.(*pp).handleMethods(0xc0003a6000, 0xc000000076, 0x1)
/Users/lowell/dev/goroot/src/fmt/print.go:603 +0x27c fp=0xc000309bd8 sp=0xc000309b48 pc=0x10c3c8c
fmt.(*pp).printArg(0xc0003a6000, 0x1532a00, 0xc00019a840, 0x76)
/Users/lowell/dev/goroot/src/fmt/print.go:686 +0x203 fp=0xc000309c70 sp=0xc000309bd8 pc=0x10c40d3
fmt.(*pp).doPrintf(0xc0003a6000, 0x15f8cb5, 0x1c, 0xc000527050, 0x3, 0x3)
/Users/lowell/dev/goroot/src/fmt/print.go:1003 +0x166 fp=0xc000309d58 sp=0xc000309c70 pc=0x10c7d26
fmt.Sprintf(0x15f8cb5, 0x1c, 0xc000527050, 0x3, 0x3, 0xc00019eaa0, 0x0)
/Users/lowell/dev/goroot/src/fmt/print.go:203 +0x66 fp=0xc000309db0 sp=0xc000309d58 pc=0x10c0ab6
github.com/hpc/kraken/core.(*ServiceLogger).Logf(0xc000148908, 0xc000309e07, 0x15f8cb5, 0x1c, 0xc000527050, 0x3, 0x3)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Logger.go:142 +0x82 fp=0xc000309e20 sp=0xc000309db0 pc=0x14318e2
github.com/hpc/kraken/core.(*EventDispatchEngine).Logf(0xc000144c00, 0xc000309f07, 0x15f8cb5, 0x1c, 0xc000527050, 0x3, 0x3)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/EventDispatchEngine.go:125 +0x6f fp=0xc000309e68 sp=0xc000309e20 pc=0x142ef6f
github.com/hpc/kraken/core.(*EventDispatchEngine).Run(0xc000144c00)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/EventDispatchEngine.go:91 +0x351 fp=0xc000309fd8 sp=0xc000309e68 pc=0x142ea11
runtime.goexit()
/Users/lowell/dev/goroot/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc000309fe0 sp=0xc000309fd8 pc=0x1058a11
created by github.com/hpc/kraken/core.(*Kraken).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Kraken.go:183 +0x86

goroutine 1 [chan receive, 9 minutes]:
main.main()
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/tmp/main.go:206 +0x1a60

goroutine 20 [chan receive]:
github.com/hpc/kraken/core.ServiceLoggerListener(0x1687120, 0xc000144a20, 0xc00009e120)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Logger.go:50 +0x66
created by github.com/hpc/kraken/core.(*Kraken).Bootstrap
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Kraken.go:152 +0x31c

goroutine 71 [chan receive, 9 minutes]:
github.com/hpc/kraken/core.(*APIServer).ServiceInit(0xc0000c38f0, 0xc000274240, 0x16872a0, 0xc00029d0d0, 0xc0000c38f0, 0x13c12eb)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:237 +0x1d8
github.com/hpc/kraken/core/proto._API_ServiceInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0002a8000, 0x1a9aac0, 0xc000226390)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1029 +0x109
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc000256180, 0xc0002a4000, 0xc000145290, 0x1a72320, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc000256180, 0xc0002a4000, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00024e030, 0xc00008f500, 0x1688260, 0xc000256180, 0xc0002a4000)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 23 [select]:
github.com/hpc/kraken/core.(*StateDifferenceEngine).Run(0xc0000b4400)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateDifferenceEngine.go:302 +0x22c
created by github.com/hpc/kraken/core.(*Kraken).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Kraken.go:184 +0xb2

goroutine 24 [select]:
github.com/hpc/kraken/core.(*StateSyncEngine).Run(0xc000124780)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:252 +0x760
created by github.com/hpc/kraken/core.(*Kraken).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Kraken.go:185 +0xde

goroutine 25 [select]:
github.com/hpc/kraken/core.(*StateMutationEngine).Run(0xc00015e000)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateMutationEngine.go:261 +0x549
created by github.com/hpc/kraken/core.(*Kraken).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Kraken.go:186 +0x10a

goroutine 26 [IO wait]:
internal/poll.runtime_pollWait(0x1f59040, 0x72, 0x0)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000136398, 0x72, 0xc0000b4100, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000136398, 0xffffffffffffff00, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000136380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000136380, 0xc00000e2b0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:238 +0x42
net.(*UnixListener).accept(0xc000144bd0, 0xc000248e80, 0xc000248e88, 0x18)
/Users/lowell/dev/goroot/src/net/unixsock_posix.go:162 +0x32
net.(*UnixListener).Accept(0xc000144bd0, 0x1610e58, 0xc00008f500, 0x1687a80, 0xc00000e2b0)
/Users/lowell/dev/goroot/src/net/unixsock.go:257 +0x47
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).Serve(0xc00008f500, 0x1684da0, 0xc000144bd0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:557 +0x210
github.com/hpc/kraken/core.(*APIServer).Run(0xc0000c38f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:331 +0xba
created by github.com/hpc/kraken/core.(*Kraken).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/Kraken.go:187 +0x136

goroutine 298 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00050a600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 297 [select]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000b4900, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00052da40, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00050a600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 35 [IO wait]:
internal/poll.runtime_pollWait(0x1f58f70, 0x72, 0x0)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001d4018, 0x72, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001d4018, 0xc0001f8000, 0x2328, 0x2328)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).ReadFrom(0xc0001d4000, 0xc0001f8000, 0x2328, 0x2328, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:219 +0x168
net.(*netFD).readFrom(0xc0001d4000, 0xc0001f8000, 0x2328, 0x2328, 0x0, 0xc000242000, 0xc00057aed0, 0x14533d8, 0xc00013c140)
/Users/lowell/dev/goroot/src/net/fd_unix.go:208 +0x5b
net.(*UDPConn).readFrom(0xc0001de000, 0xc0001f8000, 0x2328, 0x2328, 0x10356ae, 0xc00057aef8, 0x10561b0, 0xc00057af40)
/Users/lowell/dev/goroot/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFrom(0xc0001de000, 0xc0001f8000, 0x2328, 0x2328, 0xc00024e060, 0x1689280, 0xc000242000, 0xc000242000, 0x0)
/Users/lowell/dev/goroot/src/net/udpsock.go:121 +0x6d
github.com/hpc/kraken/core.(*StateSyncEngine).listen(0xc000124780, 0xc000160000, 0x1687420, 0xc0001de000)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:451 +0x9e
created by github.com/hpc/kraken/core.(*StateSyncEngine).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:215 +0x2d9

goroutine 36 [IO wait, 8 minutes]:
internal/poll.runtime_pollWait(0x1f59110, 0x72, 0x0)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000136318, 0x72, 0xc000190100, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000136318, 0xffffffffffffff00, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000136300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000136300, 0xc0001de078, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0000c0038, 0xc000168e68, 0xc000168e70, 0x18)
/Users/lowell/dev/goroot/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc0000c0038, 0x1610e58, 0xc000174600, 0x16879c0, 0xc0001de078)
/Users/lowell/dev/goroot/src/net/tcpsock.go:260 +0x47
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).Serve(0xc000174600, 0x1684d60, 0xc0000c0038, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:557 +0x210
github.com/hpc/kraken/core.(*StateSyncEngine).Run.func2(0xc000198130, 0xc000124780, 0x1684d60, 0xc0000c0038, 0xc000174600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:222 +0x43
created by github.com/hpc/kraken/core.(*StateSyncEngine).Run
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:221 +0x4ae

goroutine 294 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58c30, 0x72, 0xc000493b58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0002f8298, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0002f8298, 0xc0003b8000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0002f8280, 0xc0003b8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc0002f8280, 0xc0003b8000, 0x8000, 0x8000, 0x1a7dc60, 0x1f92ac0, 0x1d006c0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002d4028, 0xc0003b8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001da3c0, 0xc000494038, 0x9, 0x9, 0x30, 0x28, 0x158f940)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001da3c0, 0xc000494038, 0x9, 0x9, 0x9, 0xc000226a20, 0x100362e, 0xc000493e3b)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001da3c0, 0xc000494038, 0x9, 0x9, 0x139650a, 0xc000226c00, 0xc000220004)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc000494038, 0x9, 0x9, 0x167e9e0, 0xc0001da3c0, 0x0, 0x0, 0xc000226c00, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000494000, 0xc000226c00, 0xc000226c00, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00050a480, 0xc0002269f0, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc00050a480)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc00050a480)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 68 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000274040, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00009d1a0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000256180)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 69 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000256180)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 70 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58ea0, 0x72, 0xc000249b58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00024c098, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00024c098, 0xc00025a000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00024c080, 0xc00025a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc00024c080, 0xc00025a000, 0x8000, 0x8000, 0x1e, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000252000, 0xc00025a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001640c0, 0xc000272038, 0x9, 0x9, 0x18, 0x1566ea0, 0x0)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001640c0, 0xc000272038, 0x9, 0x9, 0x9, 0x1049518, 0x105a290, 0xc000249db0)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001640c0, 0xc000272038, 0x9, 0x9, 0x6, 0xc000032a88, 0xc000032a00)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc000272038, 0x9, 0x9, 0x167e9e0, 0xc0001640c0, 0x0, 0xc000000000, 0xc000249e38, 0xc000249eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000272000, 0xc0000bedc0, 0xc0000bedc0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000256180, 0xc000226120, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc000256180)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc000256180)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 77 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000254b90, 0xc000250530, 0x5, 0x5, 0xc0002447d8, 0x100dd28, 0xc0001f0346)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000254b90, 0xc000250530, 0x5, 0x5, 0x1508c96, 0x3, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0002267e0, 0xc000250530, 0x5, 0x5, 0xc0002a60c0, 0xc0001b1860, 0x13a7392)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:401 +0x55
io.ReadAtLeast(0x167f220, 0xc0002267e0, 0xc000250530, 0x5, 0x5, 0x5, 0xc000256a80, 0xc0002a4100, 0xc000000005)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167f220, 0xc0002267e0, 0xc000250530, 0x5, 0x5, 0x0, 0xc0001b19a0, 0x10c4748)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0002a4100, 0xc000250530, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:385 +0xbf
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000250520, 0x400000, 0xc0001b1a28, 0x1006e6c, 0xc00008ea80, 0x4, 0x8, 0xc00008ea80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:476 +0x65
github.com/hpc/kraken/vendor/google.golang.org/grpc.recv(0xc000250520, 0x22a4040, 0x1a9aac0, 0xc0002a4100, 0x0, 0x0, 0x15ab0e0, 0xc000254e60, 0x400000, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:605 +0x4d
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0002a80b0, 0x15ab0e0, 0xc000254e60, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/stream.go:1011 +0x122
github.com/hpc/kraken/core/proto.(*aPILoggerInitServer).Recv(0xc00029d2c0, 0xc000254b06, 0x15ea5b9, 0x5)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1112 +0x62
github.com/hpc/kraken/core.(*APIServer).LoggerInit(0xc0000c38f0, 0x1687840, 0xc00029d2c0, 0x1688b60, 0xc0000c38f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:315 +0x7d
github.com/hpc/kraken/core/proto._API_LoggerInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0002a80b0, 0x1a9aac0, 0xc000226840)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1093 +0xad
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc000256a80, 0xc0002a4100, 0xc000145290, 0x1a72380, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc000256a80, 0xc0002a4100, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00024e340, 0xc00008f500, 0x1688260, 0xc000256a80, 0xc0002a4100)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 74 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0002743c0, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001bb380, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000256a80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 75 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000256a80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 76 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58dd0, 0x72, 0xc00024bb58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00024c398, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00024c398, 0xc0002b8000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00024c380, 0xc0002b8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc00024c380, 0xc0002b8000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000252030, 0xc0002b8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001645a0, 0xc0002721f8, 0x9, 0x9, 0xbe4e77ff, 0xc000076000, 0xc00024bd48)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001645a0, 0xc0002721f8, 0x9, 0x9, 0x9, 0xc0342ebbf0, 0x342ebbf00100362e, 0x5bd75497)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001645a0, 0xc0002721f8, 0x9, 0x9, 0xbeedf305f42ebbf0, 0x1320eec, 0x1a7d1e0)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002721f8, 0x9, 0x9, 0x167e9e0, 0xc0001645a0, 0x0, 0x0, 0xc000254a68, 0xc00024beb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002721c0, 0xc0002d8000, 0xc0002d8000, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000256a80, 0xc000226600, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc000256a80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc000256a80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 289 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00052efc0, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000165a40, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000496c00)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 293 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00050a480)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 261 [select]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0001a99a0, 0xc0001ec3d0, 0x5, 0x5, 0xc000497980, 0xc000172500, 0x3d)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0001a99a0, 0xc0001ec3d0, 0x5, 0x5, 0x1508c96, 0x3, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0001a0450, 0xc0001ec3d0, 0x5, 0x5, 0xc0000d2100, 0xc00005d860, 0x13a7392)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:401 +0x55
io.ReadAtLeast(0x167f220, 0xc0001a0450, 0xc0001ec3d0, 0x5, 0x5, 0x5, 0xc000497980, 0xc000172500, 0xc000000005)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167f220, 0xc0001a0450, 0xc0001ec3d0, 0x5, 0x5, 0x0, 0x0, 0xc00008d280)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000172500, 0xc0001ec3d0, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:385 +0xbf
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0001ec3c0, 0x400000, 0x3d, 0x3d, 0x15ab0e0, 0xc000255860, 0x8, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:476 +0x65
github.com/hpc/kraken/vendor/google.golang.org/grpc.recv(0xc0001ec3c0, 0x22a4040, 0x1a9aac0, 0xc000172500, 0x0, 0x0, 0x15ab0e0, 0xc0002558b0, 0x400000, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:605 +0x4d
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0003e80b0, 0x15ab0e0, 0xc0002558b0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/stream.go:1011 +0x122
github.com/hpc/kraken/core/proto.(*aPILoggerInitServer).Recv(0xc0000a9c40, 0xc000255808, 0x15ea5b9, 0x5)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1112 +0x62
github.com/hpc/kraken/core.(*APIServer).LoggerInit(0xc0000c38f0, 0x1687840, 0xc0000a9c40, 0x1688b60, 0xc0000c38f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:315 +0x7d
github.com/hpc/kraken/core/proto._API_LoggerInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0003e80b0, 0x1a9aac0, 0xc0001a0480)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1093 +0xad
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc000497980, 0xc000172500, 0xc000145290, 0x1a72380, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc000497980, 0xc000172500, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00018ade0, 0xc00008f500, 0x1688260, 0xc000497980, 0xc000172500)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 295 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc00039a640, 0xc0002ac490, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00039a640, 0xc0002ac490, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000226bd0, 0xc0002ac490, 0x5, 0x5, 0x0, 0xc00045a860, 0x13a7392)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:401 +0x55
io.ReadAtLeast(0x167f220, 0xc000226bd0, 0xc0002ac490, 0x5, 0x5, 0x5, 0xc00050a480, 0xc000158100, 0x5)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167f220, 0xc000226bd0, 0xc0002ac490, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000158100, 0xc0002ac490, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:385 +0xbf
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002ac480, 0x400000, 0x0, 0x0, 0x0, 0x0, 0x8, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:476 +0x65
github.com/hpc/kraken/vendor/google.golang.org/grpc.recv(0xc0002ac480, 0x22a4040, 0x1a9aac0, 0xc000158100, 0x0, 0x0, 0x15ab0e0, 0xc00039a690, 0x400000, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:605 +0x4d
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc000186210, 0x15ab0e0, 0xc00039a690, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/stream.go:1011 +0x122
github.com/hpc/kraken/core/proto.(*aPILoggerInitServer).Recv(0xc00029d3f0, 0x11c65ad, 0x10, 0x18)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1112 +0x62
github.com/hpc/kraken/core.(*APIServer).LoggerInit(0xc0000c38f0, 0x1687840, 0xc00029d3f0, 0x1688b60, 0xc0000c38f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:315 +0x7d
github.com/hpc/kraken/core/proto._API_LoggerInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc000186210, 0x1a9aac0, 0xc000226c30)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1093 +0xad
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc00050a480, 0xc000158100, 0xc000145290, 0x1a72380, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc00050a480, 0xc000158100, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000216140, 0xc00008f500, 0x1688260, 0xc00050a480, 0xc000158100)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 292 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000b4680, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00052d9e0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00050a480)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 260 [chan receive]:
github.com/hpc/kraken/core.(*APIServer).MutationInit(0xc0000c38f0, 0xc000275080, 0x16871e0, 0xc0000a9b70, 0xc0000c38f0, 0x13c12eb)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:268 +0x206
github.com/hpc/kraken/core/proto._API_MutationInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0003e8000, 0x1a9aac0, 0xc0001a0240)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1050 +0x109
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc00050a600, 0xc000172400, 0xc000145290, 0x1a72340, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc00050a600, 0xc000172400, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002161a0, 0xc00008f500, 0x1688260, 0xc00050a600, 0xc000172400)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 312 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000462dc0, 0xc0001abbb0, 0x5, 0x5, 0xc000497080, 0xc000218500, 0x62)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000462dc0, 0xc0001abbb0, 0x5, 0x5, 0x153ca45, 0x8, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0001b7e90, 0xc0001abbb0, 0x5, 0x5, 0xc000192150, 0xc00020f820, 0x13a7392)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:401 +0x55
io.ReadAtLeast(0x167f220, 0xc0001b7e90, 0xc0001abbb0, 0x5, 0x5, 0x5, 0xc000497080, 0xc000218500, 0xc000000005)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167f220, 0xc0001b7e90, 0xc0001abbb0, 0x5, 0x5, 0x0, 0x0, 0xc000167500)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000218500, 0xc0001abbb0, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:385 +0xbf
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0001abba0, 0x400000, 0x62, 0x62, 0x15aafc0, 0xc0002d3630, 0x8, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:476 +0x65
github.com/hpc/kraken/vendor/google.golang.org/grpc.recv(0xc0001abba0, 0x22a4040, 0x1a9aac0, 0xc000218500, 0x0, 0x0, 0x15aafc0, 0xc000462550, 0x400000, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:605 +0x4d
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0002a8210, 0x15aafc0, 0xc000462550, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/stream.go:1011 +0x122
github.com/hpc/kraken/core/proto.(*aPIDiscoveryInitServer).Recv(0xc00019ed80, 0x1683a20, 0xc00019a8a0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1086 +0x62
github.com/hpc/kraken/core.(*APIServer).DiscoveryInit(0xc0000c38f0, 0x1687780, 0xc00019ed80, 0x1688b60, 0xc0000c38f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:293 +0x6d
github.com/hpc/kraken/core/proto._API_DiscoveryInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0002a8210, 0x1a9aac0, 0xc0001b7ef0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1067 +0xad
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc000497080, 0xc000218500, 0xc000145290, 0x1a72360, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc000497080, 0xc000218500, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00018ad10, 0xc00008f500, 0x1688260, 0xc000497080, 0xc000218500)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 300 [chan receive, 9 minutes]:
github.com/hpc/kraken/core.(*APIServer).ServiceInit(0xc0000c38f0, 0xc0000b4c40, 0x16872a0, 0xc00029d4e0, 0xc0000c38f0, 0x13c12eb)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:237 +0x1d8
github.com/hpc/kraken/core/proto._API_ServiceInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0001862c0, 0x1a9aac0, 0xc000226fc0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1029 +0x109
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc000496c00, 0xc000158200, 0xc000145290, 0x1a72320, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc000496c00, 0xc000158200, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00018acc0, 0xc00008f500, 0x1688260, 0xc000496c00, 0xc000158200)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 262 [chan receive, 9 minutes]:
github.com/hpc/kraken/core.(*APIServer).MutationInit(0xc0000c38f0, 0xc000275280, 0x16871e0, 0xc0000a9ce0, 0xc0000c38f0, 0x13c12eb)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:268 +0x206
github.com/hpc/kraken/core/proto._API_MutationInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0003e8160, 0x1a9aac0, 0xc0001a06f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1050 +0x109
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc00050aa80, 0xc000172600, 0xc000145290, 0x1a72340, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc00050aa80, 0xc000172600, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000216340, 0xc00008f500, 0x1688260, 0xc00050aa80, 0xc000172600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 97 [chan receive, 9 minutes]:
github.com/hpc/kraken/core.(*APIServer).ServiceInit(0xc0000c38f0, 0xc0000b4200, 0x16872a0, 0xc00029d240, 0xc0000c38f0, 0x13c12eb)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:237 +0x1d8
github.com/hpc/kraken/core/proto._API_ServiceInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc000186160, 0x1a9aac0, 0xc000226690)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1029 +0x109
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc0004f5680, 0xc000158000, 0xc000145290, 0x1a72320, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc0004f5680, 0xc000158000, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001c88c0, 0xc00008f500, 0x1688260, 0xc0004f5680, 0xc000158000)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 257 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0005183c0, 0xc0005039f0, 0x5, 0x5, 0xc000354600, 0xc0001f4600, 0x60)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0005183c0, 0xc0005039f0, 0x5, 0x5, 0x153ca45, 0x8, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000527aa0, 0xc0005039f0, 0x5, 0x5, 0xc0001eecc0, 0xc00030b820, 0x13a7392)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:401 +0x55
io.ReadAtLeast(0x167f220, 0xc000527aa0, 0xc0005039f0, 0x5, 0x5, 0x5, 0xc000354600, 0xc0001f4600, 0xc000000005)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167f220, 0xc000527aa0, 0xc0005039f0, 0x5, 0x5, 0x0, 0x0, 0xc0004dc800)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0001f4600, 0xc0005039f0, 0x5, 0x5, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/transport.go:385 +0xbf
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0005039e0, 0x400000, 0x38, 0x200, 0x1c0, 0x1bf, 0x8, 0x100cf95)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:476 +0x65
github.com/hpc/kraken/vendor/google.golang.org/grpc.recv(0xc0005039e0, 0x22a4040, 0x1a9aac0, 0xc0001f4600, 0x0, 0x0, 0x15aafc0, 0xc0005186e0, 0x400000, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/rpc_util.go:605 +0x4d
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0000ff760, 0x15aafc0, 0xc0005186e0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/stream.go:1011 +0x122
github.com/hpc/kraken/core/proto.(*aPIDiscoveryInitServer).Recv(0xc00052ab80, 0x1683a20, 0xc000184180, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1086 +0x62
github.com/hpc/kraken/core.(*APIServer).DiscoveryInit(0xc0000c38f0, 0x1687780, 0xc00052ab80, 0x1688b60, 0xc0000c38f0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/APIServer.go:293 +0x6d
github.com/hpc/kraken/core/proto._API_DiscoveryInit_Handler(0x15d4b60, 0xc0000c38f0, 0x1686d00, 0xc0000ff760, 0x1a9aac0, 0xc000527b00)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/proto/API.pb.go:1067 +0xad
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00008f500, 0x1688260, 0xc000354600, 0xc0001f4600, 0xc000145290, 0x1a72360, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1124 +0x8bd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc000354600, 0xc0001f4600, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1212 +0x12a1
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00018ae30, 0xc00008f500, 0x1688260, 0xc000354600, 0xc0001f4600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 254 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0001cf140, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001bb980, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0004f5680)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 255 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0004f5680)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 256 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58d00, 0x72, 0xc00045db58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000506f18, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000506f18, 0xc000150000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc000506f00, 0xc000150000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc000506f00, 0xc000150000, 0x8000, 0x8000, 0x1e, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000508158, 0xc000150000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc00052d860, 0xc000272c78, 0x9, 0x9, 0x18, 0x1566ea0, 0x0)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc00052d860, 0xc000272c78, 0x9, 0x9, 0x9, 0x1049518, 0x105a290, 0xc00045ddb0)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc00052d860, 0xc000272c78, 0x9, 0x9, 0x6, 0xc000030588, 0xc000030500)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc000272c78, 0x9, 0x9, 0x167e9e0, 0xc00052d860, 0x0, 0xc000000000, 0xc00045de38, 0xc00045deb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000272c40, 0xc0002d8180, 0xc0002d8180, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0004f5680, 0xc0005277a0, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc0004f5680)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc0004f5680)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 306 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000496c00)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 307 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58b60, 0x72, 0xc00016ab58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000137198, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000137198, 0xc0003d0000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc000137180, 0xc0003d0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc000137180, 0xc0003d0000, 0x8000, 0x8000, 0x1e, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002520f0, 0xc0003d0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001bbd40, 0xc00015e658, 0x9, 0x9, 0x18, 0x1566ea0, 0x0)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001bbd40, 0xc00015e658, 0x9, 0x9, 0x9, 0x1049518, 0x105a290, 0xc00016adb0)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001bbd40, 0xc00015e658, 0x9, 0x9, 0x6, 0xc000037488, 0xc000037400)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc00015e658, 0x9, 0x9, 0x167e9e0, 0xc0001bbd40, 0x0, 0xc000000000, 0xc00016ae38, 0xc00016aeb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00015e620, 0xc0002783e0, 0xc0002783e0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000496c00, 0xc0001b7b60, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc000496c00)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc000496c00)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 299 [IO wait]:
internal/poll.runtime_pollWait(0x1f58a90, 0x72, 0xc00045bb58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0002f8418, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0002f8418, 0xc000412000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0002f8400, 0xc000412000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc0002f8400, 0xc000412000, 0x8000, 0x8000, 0x1e, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002d4030, 0xc000412000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001da480, 0xc0004941f8, 0x9, 0x9, 0x18, 0x1566ea0, 0x0)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001da480, 0xc0004941f8, 0x9, 0x9, 0x9, 0x1049518, 0x105a290, 0xc00045bdb0)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001da480, 0xc0004941f8, 0x9, 0x9, 0x6, 0xc000032a88, 0xc000032a00)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc0004941f8, 0x9, 0x9, 0x167e9e0, 0xc0001da480, 0x0, 0xc000000000, 0xc00045be38, 0xc00045beb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0004941c0, 0xc0002d8a40, 0xc0002d8a40, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00050a600, 0xc000226d50, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc00050a600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc00050a600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 309 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00052f080, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001da8a0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000497080)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 310 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000497080)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 311 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f589c0, 0x72, 0xc00005fb58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000137318, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000137318, 0xc00042a000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc000137300, 0xc00042a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc000137300, 0xc00042a000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002520f8, 0xc00042a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0003ec000, 0xc00015e818, 0x9, 0x9, 0xceb91c00, 0xc000076800, 0xc00005fd48)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0003ec000, 0xc00015e818, 0x9, 0x9, 0x9, 0xc013b72e20, 0x13b72e200100362e, 0x5bd754a9)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0003ec000, 0xc00015e818, 0x9, 0x9, 0xbeedf30a53b72e20, 0x4119cb2ed, 0x1a7d1e0)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc00015e818, 0x9, 0x9, 0x167e9e0, 0xc0003ec000, 0x0, 0x0, 0xc000462ce8, 0xc00005feb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00015e7e0, 0xc00051a700, 0xc00051a700, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000497080, 0xc0001b7c80, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc000497080)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc000497080)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 323 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000b5100, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003ec3c0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00050aa80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 317 [select]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00052f1c0, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001dac00, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000497980)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 318 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000497980)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 319 [IO wait]:
internal/poll.runtime_pollWait(0x1f588f0, 0x72, 0xc000246b58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000137498, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000137498, 0xc00044a000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc000137480, 0xc00044a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc000137480, 0xc00044a000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000252100, 0xc00044a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0003ec1e0, 0xc00015e9d8, 0x9, 0x9, 0x1cd10c2d, 0xc0004ea400, 0xc000246d48)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0003ec1e0, 0xc00015e9d8, 0x9, 0x9, 0x9, 0xc03ad007c0, 0x3ad007c00100362e, 0x5bd756d8)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0003ec1e0, 0xc00015e9d8, 0x9, 0x9, 0xbeedf3963ad007c0, 0x865fb4a31a, 0x1a7d1e0)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc00015e9d8, 0x9, 0x9, 0x167e9e0, 0xc0003ec1e0, 0x0, 0x0, 0xc0004630f8, 0xc000246eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00015e9a0, 0xc0001a60c0, 0xc0001a60c0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000497980, 0xc00019a360, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc000497980)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc000497980)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 324 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00050aa80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 325 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58820, 0x72, 0xc000490b58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0002f8898, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0002f8898, 0xc000452000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0002f8880, 0xc000452000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc0002f8880, 0xc000452000, 0x8000, 0x8000, 0x1e, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002d4068, 0xc000452000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001dad20, 0xc0004943b8, 0x9, 0x9, 0x18, 0x1566ea0, 0x0)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001dad20, 0xc0004943b8, 0x9, 0x9, 0x9, 0x1049518, 0x105a290, 0xc000490db0)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001dad20, 0xc0004943b8, 0x9, 0x9, 0x6, 0xc00002e088, 0xc00002e000)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc0004943b8, 0x9, 0x9, 0x167e9e0, 0xc0001dad20, 0x0, 0xc000000000, 0xc000490e38, 0xc000490eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000494380, 0xc0001a6160, 0xc0001a6160, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00050aa80, 0xc000227380, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc00050aa80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc00050aa80)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 321 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00052f2c0, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001db020, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000354600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 338 [select, 9 minutes]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000354600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 339 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x1f58750, 0x72, 0xc000245b58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000137718, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000137718, 0xc000558000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc000137700, 0xc000558000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc000137700, 0xc000558000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc000252108, 0xc000558000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0003ec4e0, 0xc00015eb98, 0x9, 0x9, 0xf90319f6, 0xc000166400, 0xc000245d48)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0003ec4e0, 0xc00015eb98, 0x9, 0x9, 0x9, 0xc01eb3d148, 0x1eb3d1480100362e, 0x5bd754bf)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0003ec4e0, 0xc00015eb98, 0x9, 0x9, 0xbeedf30fdeb3d148, 0x93be6b0e3, 0x1a7d1e0)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc00015eb98, 0x9, 0x9, 0x167e9e0, 0xc0003ec4e0, 0x0, 0x0, 0xc000463328, 0xc000245eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00015eb60, 0xc000022620, 0xc000022620, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000354600, 0xc00019a5d0, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc000354600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc000354600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e

goroutine 6780 [select]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0000bb800)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xdc6

goroutine 6779 [runnable]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00021b000, 0x1, 0x0, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00014e840, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0000bb800)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xda1

goroutine 6388 [sleep]:
time.Sleep(0x2540b7575)
/Users/lowell/dev/goroot/src/runtime/time.go:105 +0x14f
github.com/hpc/kraken/core.(*StateSyncEngine).wakeForNext.func1(0xbeedf3980a28b158, 0x880be39a87, 0x1a7d1e0, 0xc000124780)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:501 +0x10d
created by github.com/hpc/kraken/core.(*StateSyncEngine).wakeForNext
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/core/StateSyncEngine.go:498 +0xf9

goroutine 6782 [runnable]:
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).Write(0xc0000bb800, 0xc000218600, 0xc000316000, 0x5, 0x160, 0xc0002eedc0, 0x150, 0x2b2, 0xc000602032, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:879 +0xfe
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).sendResponse(0xc00008f500, 0x1688260, 0xc0000bb800, 0xc000218600, 0x15b3760, 0xc00021b100, 0x0, 0x0, 0xc000602032, 0x0, ...)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:820 +0x42e
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00008f500, 0x1688260, 0xc0000bb800, 0xc000218600, 0xc000145290, 0x1a72c20, 0x0, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1004 +0x5ae
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00008f500, 0x1688260, 0xc0000bb800, 0xc000218600, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:1208 +0x1308
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0000b3d10, 0xc00008f500, 0x1688260, 0xc0000bb800, 0xc000218600)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:686 +0x9f
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:684 +0xa1

goroutine 6781 [IO wait]:
internal/poll.runtime_pollWait(0x1f58680, 0x72, 0xc000061b58)
/Users/lowell/dev/goroot/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001d5e98, 0x72, 0xffffffffffffff00, 0x1680960, 0x1a346e0)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001d5e98, 0xc000466000, 0x8000, 0x8000)
/Users/lowell/dev/goroot/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001d5e80, 0xc000466000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/internal/poll/fd_unix.go:169 +0x1d6
net.(*netFD).Read(0xc0001d5e80, 0xc000466000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00000e2b0, 0xc000466000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
/Users/lowell/dev/goroot/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc0001cdf20, 0xc00015ed58, 0x9, 0x9, 0x1ffba69b, 0xc0004ea800, 0xc000061d48)
/Users/lowell/dev/goroot/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x167e9e0, 0xc0001cdf20, 0xc00015ed58, 0x9, 0x9, 0x9, 0xc0025fd960, 0x25fd9600100362e, 0x5bd756d9)
/Users/lowell/dev/goroot/src/io/io.go:310 +0x88
io.ReadFull(0x167e9e0, 0xc0001cdf20, 0xc00015ed58, 0x9, 0x9, 0xbeedf396425fd960, 0x8662df3d88, 0x1a7d1e0)
/Users/lowell/dev/goroot/src/io/io.go:329 +0x58
github.com/hpc/kraken/vendor/golang.org/x/net/http2.readFrameHeader(0xc00015ed58, 0x9, 0x9, 0x167e9e0, 0xc0001cdf20, 0x0, 0x0, 0xc0001a2f68, 0xc000061eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/hpc/kraken/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00015ed20, 0xc00051b560, 0xc00051b560, 0x0, 0x0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/golang.org/x/net/http2/frame.go:492 +0xa3
github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0000bb800, 0xc0001d1e30, 0x1610eb0)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00008f500, 0x1688260, 0xc0000bb800)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:682 +0xdd
github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00008f500, 0x1688260, 0xc0000bb800)
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:644 +0x43
created by github.com/hpc/kraken/vendor/google.golang.org/grpc.(*Server).handleRawConn
/Users/lowell/dev/gopath/src/github.com/hpc/kraken/vendor/google.golang.org/grpc/server.go:643 +0x57e
got stream read error on mutation stream: rpc error: code = Unavailable desc = transport is closing
got stream read error on mutation stream: rpc error: code = Unavailable desc = transport is closing
pn1805641:piboot lowell$ got stream send error on logger stream: EOF

pn1805641:piboot lowell$ got stream send error on logger stream: EOF

SSE catchupSync is deeply broken

SSE catchupSync needs rewritten. It can't properly handle the case where we used to have a queue, but it's disappeared. This can lead to a deadlock.

Replace otiai10's deep copy function

Either write an in-house deep copy function (Go doesn't have a native one) or find a popularly-used one on Godoc.

Add ability to set starting state from a file

Ideally both cfg and dsc. This will make resuming a kraken easier as well as systemd style starting.

example fails if vbox hasn't initialized interface

If this is the first VM starting on the vbox network used in the example, vbox will not have initialized the interface yet. This should not cause the example to fail.

Dashboard Refresh of a Node Page Results in 404

If you refresh the browser on a node page like http://192.168.57.10/node/123E4567-E89D-12D3-A456-425455440096, it results in a 404.

192.168.57.1 - - [16/Feb/2019:22:27:34 +0000] "GET / HTTP/1.1" 200 2064 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15" "-"
192.168.57.1 - - [16/Feb/2019:22:27:34 +0000] "GET /node/undefined HTTP/1.1" 404 3650 "http://192.168.57.10/node/123E4567-E89D-12D3-A456-425455440096" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15" "-"
192.168.57.1 - - [16/Feb/2019:22:27:41 +0000] "GET /node/123E4567-E89D-12D3-A456-425455440096 HTTP/1.1" 404 3650 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0.3 Safari/605.1.15" "-"

Service stopping is broken

Setting a service state to "STOP" doesn't seem actually stop the service in some cases.

The ServiceManager code is old and broken and could use an overhaul.

Export mutation graph in json/dot syntax through restapi

Could be done by reworking DumpGraph and adding a hook in restapi.

This would allow easy inspection and debugging of the mutation graph.

Create .spec file for RPM builds

...should allow passing the build config as an RPM macro.

Update/create unit tests for core/

Most of the existing unit tests don't work anymore. There's very little test coverage right now.

Add flag to start kraken with the SME frozen

Something simple like -freeze.

Implement IPMI module

Need IPMI v1.5 & v2.0

val.IsZero Undefined in core/StateMutation.go

When building kraken using kraken-build from the master branch, the following error occurs:

$ ./kraken-build -v
2019/12/23 21:57:07 using kraken at: /home/devon/test
2019/12/23 21:57:07 setting up build environment
2019/12/23 21:57:07 executing template: includes.go.tpl
2019/12/23 21:57:07 executing template: main.go.tpl
2019/12/23 21:57:07 executing template: pprof.go.tpl
2019/12/23 21:57:07 building: linux-amd64 (GOOS: linux, GOARCH; amd64)
2019/12/23 21:57:07 Run: go build -o main includes.go main.go pprof.go
# github.com/hpc/kraken/core
../core/StateMutation.go:103:9: val.IsZero undefined (type reflect.Value has no field or method IsZero)
../core/StateMutation.go:154:9: val.IsZero undefined (type reflect.Value has no field or method IsZero)
note: module requires Go 1.13
2019/12/23 21:57:08 failed to build linux-amd64: exit status 2
2019/12/23 21:57:08 building: darwin-amd64 (GOOS: darwin, GOARCH; amd64)
2019/12/23 21:57:08 Run: go build -o main includes.go main.go pprof.go
# github.com/hpc/kraken/core
../core/StateMutation.go:103:9: val.IsZero undefined (type reflect.Value has no field or method IsZero)
../core/StateMutation.go:154:9: val.IsZero undefined (type reflect.Value has no field or method IsZero)
note: module requires Go 1.13
2019/12/23 21:57:09 failed to build darwin-amd64: exit status 2

Tested with a fresh git clone of kraken outside of GOPATH.

Address the way kraken calls services

Currently, the way that kraken launches services is that it calls kraken with an argv[0] of the format [kraken:<service_ID>]; however, since u-root uses a single BusyBox binary with symlinks for each command, it will not be able to "find" kraken when doing this since the binary uses argv[0] to determine what command was called.

We should consider modifying this mechanism to avoid hack solutions when generating u-root-compatible kraken source trees (e.g. modifying the service-launching code during generation).

Comparing the output files obtained from metaphlan2 with kraken2

I ran one sample through Metaphlan2 and kraken2 for classification. But the results from Metaphlan2 and Kraken2 is completely different. I am not sure which one I have to rely? Any insight would be greatly appreciated

Module API Simplification/kraken-build should auto-generate service instance code

kraken-build should use the YAML config to generate service instance code, reducing module code requirements.

Implement Redfish module

...

Websocket/restapi doesn't stop trying to send graph updates...

...even after the connection as died. Probably not great. Running with -log 9 will show this.

restapi race sometimes doesn't start on correct IP

restapi appears to start the http instance before the -ipapi ip gets passed on some portion of the time, and doesn't restart http to register the change.

Incorrect url in pxe module

Is the url in line 204 supposed to be "url2"?

https://github.com/hpc/kraken/blob/d8359801117ee8b6e9def71def6161c7cdc25f19/modules/pxe/dhcp.go#L192-L211

Note: this also exists in pipxe module at line 318 in dhcp.go

Silently fails to get node cfg info from mutation event

In the pipower module, me.NodeCfg.GetValues will return a map with two empty strings randomly. This causes nodes to get stuck in hang state. I've tried to debug this, but couldn't find an obvious problem with the underlying functions.

func (pp *PiPower) handleMutation(m lib.Event) {
	if m.Type() != lib.Event_STATE_MUTATION {
		pp.api.Log(lib.LLINFO, "got an unexpected event type on mutation channel")
	}
	me := m.Data().(*core.MutationEvent)
	vs, err := me.NodeCfg.GetValues([]string{ChassisURL, RankURL})
        // Debugging...
	if vs[ChassisURL].String() == "" {
		panic(fmt.Sprintf("chassisurl is empty nodecfg: %#v extension: %#v", me.NodeCfg, me.NodeCfg.GetExtensionURLs()))
	}

Create VirtualBox example for u-root

Similar to the normal VBox example, it would be useful to have one for Kraken's firmware implementation in u-root.

This assumes #100 is solved and #134 is merged.

layer1 epel.repo file has an unreachable repo

Similar to issue #16 - utils/layer1/centos-supervisord/epel.repo has an unreachable repo in it:

baseurl=http://nmc-repo.newmexicoconsortium.org/centos/7/epel/$basearch

I'll provide a patch shortly.

Mutation excludes don't work properly...

This has to do with how we derive the epistemology of the graph. This needs a graph epistemology rethink.

Add CI Integration

The project should have CI based checks integrated. I've had good luck with CircleCI, but I can certainly help with any other preferred tooling.

A reasonable example with CircleCI: https://github.com/cfg8er/cfg8er/blob/master/.circleci/config.yml.

#74 and #73 were the start of some initial work towards this.

TODOs:

Existing tests passing (#75)
Golint (or equiv. like revive) passing. Primarily appears to be exported functions, const, etc without comments currently.
Add CI Config
Enable CI integration and enforce branch protection on the Github project

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble