GithubHelp home page GithubHelp logo

goprocess's Introduction

goprocess - lifecycles in go

travisbadge

(Based on https://github.com/jbenet/go-ctxgroup)

goprocess introduces a way to manage process lifecycles in go. It is much like go.net/context (it actually uses a Context), but it is more like a Context-WaitGroup hybrid. goprocess is about being able to start and stop units of work, which may receive Close signals from many clients. Think of it like a UNIX process tree, but inside go.

goprocess seeks to minimally affect your objects, so you can use it with both embedding or composition. At the heart of goprocess is the Process interface:

// Process is the basic unit of work in goprocess. It defines a computation
// with a lifecycle:
// - running (before calling Close),
// - closing (after calling Close at least once),
// - closed (after Close returns, and all teardown has _completed_).
//
// More specifically, it fits this:
//
//   p := WithTeardown(tf) // new process is created, it is now running.
//   p.AddChild(q)         // can register children **before** Closing.
//   go p.Close()          // blocks until done running teardown func.
//   <-p.Closing()         // would now return true.
//   <-p.childrenDone()    // wait on all children to be done
//   p.teardown()          // runs the user's teardown function tf.
//   p.Close()             // now returns, with error teardown returned.
//   <-p.Closed()          // would now return true.
//
// Processes can be arranged in a process "tree", where children are
// automatically Closed if their parents are closed. (Note, it is actually
// a Process DAG, children may have multiple parents). A process may also
// optionally wait for another to fully Close before beginning to Close.
// This makes it easy to ensure order of operations and proper sequential
// teardown of resurces. For example:
//
//   p1 := goprocess.WithTeardown(func() error {
//     fmt.Println("closing 1")
//   })
//   p2 := goprocess.WithTeardown(func() error {
//     fmt.Println("closing 2")
//   })
//   p3 := goprocess.WithTeardown(func() error {
//     fmt.Println("closing 3")
//   })
//
//   p1.AddChild(p2)
//   p2.AddChild(p3)
//
//
//   go p1.Close()
//   go p2.Close()
//   go p3.Close()
//
//   // Output:
//   // closing 3
//   // closing 2
//   // closing 1
//
// Process is modelled after the UNIX processes group idea, and heavily
// informed by sync.WaitGroup and go.net/context.Context.
//
// In the function documentation of this interface, `p` always refers to
// the self Process.
type Process interface {

  // WaitFor makes p wait for q before exiting. Thus, p will _always_ close
  // _after_ q. Note well: a waiting cycle is deadlock.
  //
  // If q is already Closed, WaitFor calls p.Close()
  // If p is already Closing or Closed, WaitFor panics. This is the same thing
  // as calling Add(1) _after_ calling Done() on a wait group. Calling WaitFor
  // on an already-closed process is a programming error likely due to bad
  // synchronization
  WaitFor(q Process)

  // AddChildNoWait registers child as a "child" of Process. As in UNIX,
  // when parent is Closed, child is Closed -- child may Close beforehand.
  // This is the equivalent of calling:
  //
  //  go func(parent, child Process) {
  //    <-parent.Closing()
  //    child.Close()
  //  }(p, q)
  //
  // Note: the naming of functions is `AddChildNoWait` and `AddChild` (instead
  // of `AddChild` and `AddChildWaitFor`) because:
  // - it is the more common operation,
  // - explicitness is helpful in the less common case (no waiting), and
  // - usual "child" semantics imply parent Processes should wait for children.
  AddChildNoWait(q Process)

  // AddChild is the equivalent of calling:
  //  parent.AddChildNoWait(q)
  //  parent.WaitFor(q)
  AddChild(q Process)

  // Go creates a new process, adds it as a child, and spawns the ProcessFunc f
  // in its own goroutine. It is equivalent to:
  //
  //   GoChild(p, f)
  //
  // It is useful to construct simple asynchronous workers, children of p.
  Go(f ProcessFunc) Process

  // Close ends the process. Close blocks until the process has completely
  // shut down, and any teardown has run _exactly once_. The returned error
  // is available indefinitely: calling Close twice returns the same error.
  // If the process has already been closed, Close returns immediately.
  Close() error

  // Closing is a signal to wait upon. The returned channel is closed
  // _after_ Close has been called at least once, but teardown may or may
  // not be done yet. The primary use case of Closing is for children who
  // need to know when a parent is shutting down, and therefore also shut
  // down.
  Closing() <-chan struct{}

  // Closed is a signal to wait upon. The returned channel is closed
  // _after_ Close has completed; teardown has finished. The primary use case
  // of Closed is waiting for a Process to Close without _causing_ the Close.
  Closed() <-chan struct{}
}

goprocess's People

Contributors

jbenet avatar stebalien avatar rht avatar whyrusleeping avatar cryptix avatar

Stargazers

Md Talha Zubayer avatar  avatar Patrick Organ avatar Oleg Mysin avatar  avatar Al Liu avatar Javed Khan avatar Thomas Jay Rush avatar Matt Ma avatar 虫子樱桃 avatar godcong avatar  avatar Eclésio Junior avatar venjiang avatar Guilherme Marcial avatar Nikita avatar tdakkota avatar yaoci shen avatar SergeyRadist avatar rxw avatar Osinniy avatar Vamshik Shetty avatar Advaith Doosa avatar  avatar Terence avatar John Yellow avatar Govind Mohan avatar Bogdan Dinu avatar Glen Keane avatar  avatar A ghost. avatar caivega avatar yang chengkai avatar Drew O'Meara avatar Doru Carastan avatar Jim Coggeshall avatar Park Sang kil avatar Hlib Kanunnikov avatar BenDerPan avatar Paul avatar Adam Fowler avatar  avatar Nex Zhu avatar  avatar xiaohuo avatar qiwaa avatar  avatar xhliu avatar  avatar Alex Grin avatar Mobin Ho avatar Lex Tang avatar  avatar Adrian Lanzafame avatar Fotis Nikolaidis avatar Wolfmetr avatar Jonhnny Weslley avatar Angus H. avatar  avatar Xiaohan Song avatar Omer Katz avatar Fredrik Forsmo avatar  avatar  avatar Taichi Nakashima avatar Quinn Slack avatar Ken Keiter avatar NDuma avatar Thomas Sileo avatar Athan avatar

Watchers

 avatar  avatar James Cloos avatar  avatar  avatar

goprocess's Issues

semantics of AddChild() after Close()

Ran into an interesting case. suppose:

fire := make(chan struct{})
handler := func(proc goprocess.Process) { ... }
p := goprocess.Go(func(proc goprocess.Process) {
  for {
    select {
    case <-fire:
      <-proc.Go(handler).Closed()
    case <-proc.Closing():
      return
    }
  }
})

go p.Close()
fire <- struct{}{} 
  • order of <-proc.Closing() and <-fire is not determined
  • so proc.Go(handler) (proc.AddChild(q)) races with p.Close() and may deadlock

I can fix this example with:

fire := make(chan struct{})
handler := func(proc goprocess.Process) { ... }
p := goprocess.Go(func(proc goprocess.Process) {
  for {
    select {
    case <-fire:

      // really make sure we're not closing.
      select {
      case <-proc.Closing():
        return
      default:
      }
      <-proc.Go(handler).Closed()

    case <-proc.Closing():
      return
    }
  }
})

go p.Close()
fire <- struct{}{} 

But I wonder whether the semantics of AddChild while Closing() should be:

  • it is always an error and may deadlock, or
  • it is fine to add children while closing. these are immediately signaled to close.

weird case with nil channel

goroutine 22 [chan receive (nil chan)]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context.func·001()
    /Users/jbenet/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context/context.go:31 +0x65
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context.WithContextAndTeardown
    /Users/jbenet/go/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context/context.go:33 +0x172

various failures in multicore configuration

Running go test --cpu 4 with or without --race results in lots of different failures.

@jbenet These cause clients to deadlock.

  • [x]
goprocess (fix-424-blockservice-async-HasBlock) λ. go test ./... --cpu 4 --race                                                                                                     1
--- FAIL: TestCloseAfterChildren-4 (0.00s)
        goprocess_test.go:486: none should be closed
FAIL
FAIL    github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess     1.282s
?       github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context     [no test files]
ok      github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/ratelimit   0.016s
  • [x]
goprocess (fix-424-blockservice-async-HasBlock) λ. go test ./... --cpu 4 --race                                                                                                     1
--- FAIL: TestOnClosed-4 (0.00s)
        goprocess_test.go:498: context not in group: 0 [00 01 10 11]
        goprocess_test.go:498: context not in group: 11 [0 1]
--- FAIL: TestCloseAfterChildren-4 (0.00s)
        goprocess_test.go:486: none should be closed
FAIL
FAIL    github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess     1.283s
?       github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/context     [no test files]
ok      github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/ratelimit   0.017s
  • [x]
--- FAIL: TestRateLimitGoDoesntBlock-4 (0.00s)
        ratelimit_test.go:65: create a rate limiter with limit of 3
        ratelimit_test.go:71: spawn 6 children with usual Process.Go.
        ratelimit_test.go:83: should not have blocked.
        ratelimit_test.go:78: spawned 0
        ratelimit_test.go:78: spawned 1
        ratelimit_test.go:78: spawned 2
        ratelimit_test.go:78: spawned 3
        ratelimit_test.go:78: spawned 4
        ratelimit_test.go:78: spawned 5
        ratelimit_test.go:88: process.Go blocked. it should not.
        ratelimit_test.go:91: drain children so they close
        ratelimit_test.go:94: closed 0
        ratelimit_test.go:94: closed 1
        ratelimit_test.go:94: closed 2
        ratelimit_test.go:94: closed 3
        ratelimit_test.go:94: closed 4
        ratelimit_test.go:94: closed 5
FAIL
FAIL    github.com/jbenet/go-ipfs/Godeps/_workspace/src/github.com/jbenet/goprocess/ratelimit   0.017s

custom context leads to extra goroutine per proc

While debugging an issue in our relay infrastructure, I found ~300,000 goroutines with this stacktrace:

goroutine 2828669022 [select]:
context.propagateCancel.func1(0xf780c0, 0xc0002507b0, 0xf6d320, 0xc496a5b3e0)
#011/usr/local/go/src/context/context.go:259 +0xd8
created by context.propagateCancel
#011/usr/local/go/src/context/context.go:258 +0x18a

context.propagateCancel is responsible for watching the parent context and cancelling the child when the parent is cancelled. If the parent is a timed context or a cancellable context, it'll insert a hook in the parent cancel functions to be notified.

However, if the context is a custom one, a background context, or a TODO context, it'll spawn a dedicated goroutine to watch the parent and trigger the child accordingly.


I'm not entirely sure go-process is causing this build-up for us (we could be using background contexts), but I needed a place to file this discovery 😛

panic: assignment to entry in nil map

I have recently started getting these occasionally. I have also recently updated deps so that I started using the v0.1.2 tag.

@Stebalien I see you merged some recent fixes, might this be related? I never saw this panic before.

=== RUN   TestConsensusPin
panic: assignment to entry in nil map

goroutine 93 [running]:
github.com/jbenet/goprocess.(*process).AddChildNoWait(0xc000231a40, 0x113fbc0, 0xc0000b3e60)
        /home/hector/go/pkg/mod/github.com/jbenet/[email protected]/impl-mutex.go:72 +0x18d
github.com/jbenet/goprocess/context.WithProcessClosing(0x112eaa0, 0xc000218ac0, 0x113fbc0, 0xc000231a40, 0x0, 0x0)
        /home/hector/go/pkg/mod/github.com/jbenet/[email protected]/context/context.go:85 +0x170
github.com/jbenet/goprocess/context.OnClosingContext(...)
        /home/hector/go/pkg/mod/github.com/jbenet/[email protected]/context/derive.go:12
github.com/ipfs/go-bitswap.(*Bitswap).provideWorker(0xc0002ea500, 0x113fbc0, 0xc000231a40)
        /home/hector/go/pkg/mod/github.com/ipfs/[email protected]/workers.go:109 +0x6a
github.com/jbenet/goprocess.(*process).Go.func1(0xc0004ba600, 0xc000231a40, 0xc000231aa0)
        /home/hector/go/pkg/mod/github.com/jbenet/[email protected]/impl-mutex.go:112 +0x3c
created by github.com/jbenet/goprocess.(*process).Go
        /home/hector/go/pkg/mod/github.com/jbenet/[email protected]/impl-mutex.go:111 +0x1f7

leak

this has a leak thanks to the Background() process. it's not big, but it is there.

TestGoClosing blinks

Hi,

What is the actual expected behaviour of this portion of the code:

		a.Go(func(p Process) {

			select {
			case <-p.Closing():
				// p should be marked as closing
			default:
				t.Error("not marked closing when it should be.")
			}

			ready <- struct{}{}
		})

?

It seems like in general it should not work because "p" that is passed to ProcessFunc is actually a new process and it doesn't inherit "closing" state immediately.

I'm trying to deal with goroutine leaks in IPFS tests with "-race" flag enabled (bc there's 8192 goroutine limit in race mode) and most of the goroutines I see in stacktraces are coming from this code. Not sure yet that this is the root cause just wanted to clarify if it "should be" or "must be"?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.