Fix Sessions to properly manage worker routines about cayley

[barakmich]

I believe this is the same issue that #100 got around for the integration test, but should be stomped out once and for all

[kortschak]

@scampi Would you please post which version of cayley you were running when this happened and what configuration you were using?

[0xAX]

+1 reproduced the same bug on cayley from master

[kortschak]

The issue here is that kill chan persists over the course of the session. This means that there is a race that allows a brief window when s.kill has been replaced and an old func·011 has not yet fired the close signal. The kill chan can then be closed and a new func·011 is started with a now closed chan. We could engineer something around this, but I think it's probably best to just use tomb since the engineering there has already been done: http://godoc.org/gopkg.in/tomb.v2 (3-clause BSD-like).

[barakmich]

Tomb SGTM. We entomb the wait-and-kill goroutine and kill the killer on success? (As violent as that sounds)

[kortschak]

I need to read what Gustavo suggests, but we should be able to have a type that encapsulates the otto session. It should be less invasive than having to kill everyone involved.

[kortschak]

I've just had a look into this and it's more complicated to use tomb than I though for this case. However, if my diagnosis is correct above, the following code should fix it. @scampi and @0xAX would you try to reproduce with gremlin.(*Session).runUnsafe replaced with the following:

func (s *Session) runUnsafe(input interface{}) (otto.Value, error) {
    defer func() {
        if r := recover(); r != nil {
            if r == ErrKillTimeout {
                s.err = ErrKillTimeout
                return
            }
            panic(r)
        }
    }()

    // Use buffered chan to prevent blocking.
    s.env.Interrupt = make(chan func(), 1)

    ready := make(chan struct{})
    s.kill = make(chan struct{})
    if s.timeout >= 0 {
        go func() {
            time.Sleep(s.timeout)
            <-ready
            close(s.kill)
            s.envLock.Lock()
            defer s.envLock.Unlock()
            if s.env != nil {
                s.env.Interrupt <- func() {
                    panic(ErrKillTimeout)
                }
                s.env = s.emptyEnv
            }
        }()
    }

    s.envLock.Lock()
    env := s.env
    s.envLock.Unlock()
    close(ready)
    return env.Run(input)
}

[0xAX]

Hello @kortschak,

applied your patch, the same crash for me

[0xAX]

btw @kortschak i'm very interesting in cayley and contributing to it, now i'm reading source code of it and try to fix current bug and have a one question. How to print something custom in repl for debugging? I see that everywhere is fmt.Printf for output but i tried to put fmt.Printf in runUnsafe and go build cayley but nothing printed.

so your patch can be fix for this bug, but seems that changes doesn't apply after go build

[kortschak]

FIrst, let's get the obvious out of the way. I'm assuming you built and installed the changed code.

Can you post a gist with the output of `git diff' from within the cayley repo when you have your changes in place?

[kortschak]

Ian this is why I shouldn't do anything when I'm sick. I see you are using go build. For the sake of simplicity in explanation here, don't do that. go build does a build, and unless the object is an executable, the result is discarded, so unless you are doing go build in the cayley directory, the effect is just a check for compilability. Use go install, from within the cayley directory - because of the speed of compilation, clobbering your installed cayley should not be an issue.

[barakmich]

I tried it too -- no dice. The simple way to reproduce is to run the sample memstore with -timeout 5s -- run g.V().All() (give it something to do) and up-enter whenever you get a prompt.

The problem being, much like s.env, used to be, there's a couple goroutines then floating about pointing at s.kill

There's https://github.com/barakmich/cayley/commits/replcrash with my go at it that seems to help.

[kortschak]

I'm afraid that's racey. This is the kind of thing I was trying to avoid with my fix, but I'm not firing on all cylinders right now.

The issue I think comes down to a conflict of contractual expectations. A session is only contractually expected to run once, but in the REPL we give it a number of snippets of code to run. What needs to happen is that each snippet gets it's own killer. So probably it needs to be organised as a Session with workers that each only can be run once.

[0xAX]

Ian this is why I shouldn't do anything when I'm sick. I see you are using go build. For the sake of simplicity in explanation here, don't do that. go build does a build, and unless the object is an executable, the result is discarded, so unless you are doing go build in the cayley directory, the effect is just a check for compilability. Use go install, from within the cayley directory - because of the speed of compilation, clobbering your installed cayley should not be an issue.

Yes, i tried go build in cayley directory. Now look:

For example i just added fmt.Println(s.timeout) before https://github.com/google/cayley/blob/master/query/gremlin/session.go#L128, after it i executed go install and it nothing print after starting repl

[kortschak]

Works for me using both methods.

[0xAX]

ah @kortschak sorry for this, found a problem.

[0xAX]

@kortschak, @barakmich tested cayley with c1ff6ce and can't reproduce this bug

[barakmich]

And I'm having trouble recreating the race. The first race I got was actually in peterh/liner.

Meanwhile, I'm going to merge my patch, to halt the bleeding, because it's really bad right now.

I'm not saying it's fixed by any means; I'm leaving this bug open and changing its scope to be "Fix Sessions to properly manage worker routines", which is the correct fix as @kortschak mentions.

[kortschak]

Yeah, no worries. I have a plan for a fix, but I have not had the time to work it up yet.

Do you have the race warning for liner?

[barakmich]

Happened to be in my scrollback:

cayley> ==================
WARNING: DATA RACE
Read by goroutine 9:
  github.com/peterh/liner.(*State).WriteHistory()
      /home/barak/src/cayley/src/github.com/peterh/liner/common.go:61 +0x6c
  github.com/google/cayley/db.persist()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:199 +0x25a
  github.com/google/cayley/db.func·002()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:173 +0x195

Previous write by main goroutine:
  [failed to restore the stack]

Goroutine 9 (running) created at:
  github.com/google/cayley/db.terminal()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:182 +0xe2
  github.com/google/cayley/db.Repl()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:86 +0x188
  main.main()
      /fs/home/barak/src/cayley/src/github.com/google/cayley/cayley.go:164 +0x13b8
==================
==================
WARNING: DATA RACE
Read by goroutine 9:
  github.com/peterh/liner.(*State).WriteHistory()
      /home/barak/src/cayley/src/github.com/peterh/liner/common.go:61 +0xce
  github.com/google/cayley/db.persist()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:199 +0x25a
  github.com/google/cayley/db.func·002()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:173 +0x195

Previous write by main goroutine:
  [failed to restore the stack]

Goroutine 9 (running) created at:
  github.com/google/cayley/db.terminal()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:182 +0xe2
  github.com/google/cayley/db.Repl()
      /home/barak/src/cayley/src/github.com/google/cayley/db/repl.go:86 +0x188
  main.main()
      /fs/home/barak/src/cayley/src/github.com/google/cayley/cayley.go:164 +0x13b8
==================

[kortschak]

OK, so that must have happened in response to a ^C.

I can see why it's there, and I am causing it, but I can't see a way to fix it.

I will file an issue against liner to get @peterh's view.

[barakmich]

Yeah, it did, actually.

[peterh]

Hmm. At a guess, you're trying to save history from a goroutine that is responding to Ctrl-C while another goroutine is still inside Prompt(). That's not supported. Or, at least, it wasn't. Could you please give peterh/liner@16acdab a spin, and let me know if that fixes things?

[kortschak]

Thanks Peter, that was my feeling, and your fix looks exactly like what I was thinking. I will try it when I get to work.

[kortschak]

@peterh, that fix works as expected, thanks.