Problems on connect about elastic

[olivere]

Hi @bmelton. It's always been a problem with IP addresses up until now, so I guess that it's the same with you. No worries, we'll get this to work for you as well :-)

If you don't use any special configurations, elastic will use the default IP (127.0.0.1:9200) to sniff all nodes on the ES cluster. It does that by using the network info API, similar to the one you used in your connect.py. It parses the IP addresses of all nodes ("http_address" : "inet[/172.17.0.31:9200]",), periodically checks them for availability and adds new nodes as they come. The process is described in the Wiki. See also: Connection problems.

If you disable sniffing, you must set the IP addresses of all nodes manually by SetURL(...). Notice that elastic would not find new nodes in your cluster then.

So if you've got only one ES node in your cluster and it's available at 192.168.99.100:9200, this is supposed to work:

client, err := elastic.NewClient(
    elastic.SetSniff(false),
    elastic.SetURL("http://192.168.99.100:9200"),
)

Cheers.

[olivere]

@DasHaus I ran a program over night in the hopes of finding your problem. It simple connected to ES (which starts the sniffing process), then waits a few milliseconds, rinse and repeat. I'm afraid, I didn't have a single error in over 1 mio. connection attempts.

I'm also using ES locally on 127.0.0.1:9200 when developing and I didn't have that error once. So either I'm really lucky or there's something different between the way you are using ES/Elastic and I'm using it. Well, I'm at 1.5.0 and you're at 1.3.4, but that shouldn't be a problem. Go is 1.4.2 here.

Do you create one client for your app, or are you creating clients on demand?

[emilgpa]

I create one client for my app. For me, it's stranger, not always occurs... but when i do a change in my app, the server restarts and is there when, sometimes, occurs this error...

[olivere]

I can't get it to fail, even after 2 days. Can you run this gist for a while? If it fails, can you return the stack trace. Thank you.

[emilgpa]

Ok. I ran by go run <name_gist_file> and within seconds, the following error is raised:

no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x9d701]

goroutine 1 [running]:
github.com/olivere/elastic.(*PingService).Do(0xc20803b1a0, 0x0, 0x0, 0x0, 0x0)
    /Users/emilio/go/src/github.com/olivere/elastic/ping.go:102 +0x471
github.com/olivere/elastic.(*Client).ElasticsearchVersion(0x0, 0x405290, 0x15, 0x0, 0x0, 0x0, 0x0)
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:827 +0x9b
main.main()
    /Users/emilio/Dropbox/test_es.go:28 +0x241

goroutine 5 [select]:
net/http.(*persistConn).roundTrip(0xc208086000, 0xc20800adb0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208038090, 0xc2080320d0, 0xc20803b020, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc2080320d0, 0x687df0, 0xc208038090, 0x21, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x5b7be0, 0xc2080320d0, 0x21, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x5b7be0, 0xc2080320d0, 0x483620, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x5b7be0, 0xc2080320d0, 0xc20803afc0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:174 +0xa4
github.com/olivere/elastic.(*Client).sniffNode(0xc208080000, 0xc20801e960, 0x15, 0x0, 0x0, 0x0)
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:543 +0x16a
github.com/olivere/elastic.func·014(0xc20801e960, 0x15)
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x47
created by github.com/olivere/elastic.(*Client).sniff
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x744

goroutine 8 [IO wait]:
net.(*pollDesc).Wait(0xc208010df0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010df0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208010d90, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x687b48, 0xc20800ae28)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208036080, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x688fb8, 0xc208036080, 0xc208086058, 0xc20800f000, 0x1000, 0x1000, 0x676010, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20801ea80, 0xc20800f000, 0x1000, 0x1000, 0xc207ffc9f7, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20805a600)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20805a600, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208086000)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc208086000)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc
exit status 2

[olivere]

Can you run with go run -race .go and send the output on failure, please?

Am 02.04.2015 um 20:13 schrieb dashaus [email protected]:

Ok. I ran by go run <name_gist_file> and within seconds, the following error is raised:

no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x9d701]

goroutine 1 [running]:
github.com/olivere/elastic.(_PingService).Do(0xc20803b1a0, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/ping.go:102 +0x471
github.com/olivere/elastic.(_Client).ElasticsearchVersion(0x0, 0x405290, 0x15, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:827 +0x9b
main.main()
/Users/emilio/Dropbox/test_es.go:28 +0x241

goroutine 5 [select]:
net/http.(_persistConn).roundTrip(0xc208086000, 0xc20800adb0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(_Transport).RoundTrip(0xc208038090, 0xc2080320d0, 0xc20803b020, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc2080320d0, 0x687df0, 0xc208038090, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(_Client).send(0x5b7be0, 0xc2080320d0, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(_Client).doFollowingRedirects(0x5b7be0, 0xc2080320d0, 0x483620, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(_Client).Do(0x5b7be0, 0xc2080320d0, 0xc20803afc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/olivere/elastic.(_Client).sniffNode(0xc208080000, 0xc20801e960, 0x15, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:543 +0x16a
github.com/olivere/elastic.func·014(0xc20801e960, 0x15)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x47
created by github.com/olivere/elastic.(*Client).sniff
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x744

goroutine 8 [IO wait]:
net.(_pollDesc).Wait(0xc208010df0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc208010df0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).Read(0xc208010d90, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x687b48, 0xc20800ae28)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(_conn).Read(0xc208036080, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x688fb8, 0xc208036080, 0xc208086058, 0xc20800f000, 0x1000, 0x1000, 0x676010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(_noteEOFReader).Read(0xc20801ea80, 0xc20800f000, 0x1000, 0x1000, 0xc207ffc9f7, 0x0, 0x0)
:125 +0xd4
bufio.(_Reader).fill(0xc20805a600)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(_Reader).Peek(0xc20805a600, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(_persistConn).readLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 9 [select]:
net/http.(_persistConn).writeLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(_Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
exit status 2
—
Reply to this email directly or view it on GitHub.

[emilgpa]

no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0xc8da5]

goroutine 1 [running]:
github.com/olivere/elastic.(*PingService).Do(0xc20803acf0, 0x0, 0x0, 0x0, 0x0)
    /Users/emilio/go/src/github.com/olivere/elastic/ping.go:102 +0x565
github.com/olivere/elastic.(*Client).ElasticsearchVersion(0x0, 0x524eb0, 0x15, 0x0, 0x0, 0x0, 0x0)
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:827 +0xcf
main.main()
    /Users/emilio/Dropbox/test_es.go:28 +0x2c0

goroutine 3822 [select]:
net/http.(*persistConn).roundTrip(0xc208086000, 0xc20800aad0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1082 +0xa69
net/http.(*Transport).RoundTrip(0xc208030000, 0xc2080324e0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:235 +0x6b8
net/http.send(0xc2080324e0, 0x11bcd18, 0xc208030000, 0xc20803ac00, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:219 +0x6f3
net/http.(*Client).send(0x725000, 0xc2080324e0, 0x21, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:142 +0x201
net/http.(*Client).doFollowingRedirects(0x725000, 0xc2080324e0, 0x5c06c0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:367 +0xf18
net/http.(*Client).Do(0x725000, 0xc2080324e0, 0xc20803ab40, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:174 +0xc4
github.com/olivere/elastic.(*Client).sniffNode(0xc2080d6000, 0xc2080f71c0, 0x15, 0x0, 0x0, 0x0)
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:543 +0x19b
github.com/olivere/elastic.func·014(0xc2080f71c0, 0x15)
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x68
created by github.com/olivere/elastic.(*Client).sniff
    /Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x95f

goroutine 11 [IO wait]:
net.(*pollDesc).Wait(0xc208010370, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x63
net.(*pollDesc).WaitRead(0xc208010370, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x51
net.(*netFD).Read(0xc208010310, 0xc208089000, 0x1000, 0x1000, 0x0, 0x11bcb48, 0xc20800aea0)
    /usr/local/go/src/net/fd_unix.go:242 +0x4b3
net.(*conn).Read(0xc208036068, 0xc208089000, 0x1000, 0x1000, 0x3a0e2, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0x125
net.(*TCPConn).Read(0xc208036068, 0xc208089000, 0x1000, 0x1000, 0x5c08d0, 0x0, 0x0)
    <autogenerated>:36 +0x86
net/http.noteEOFReader.Read(0x11bdee0, 0xc208036068, 0xc208086108, 0xc208089000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x7c
net/http.(*noteEOFReader).Read(0xc20801e840, 0xc208089000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:125 +0xfc
bufio.(*Reader).fill(0xc2080526c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x382
bufio.(*Reader).Peek(0xc2080526c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0x199
net/http.(*persistConn).readLoop(0xc2080860b0)
    /usr/local/go/src/net/http/transport.go:842 +0xff
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0x10d4

goroutine 8 [IO wait]:
net.(*pollDesc).Wait(0xc208010e60, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x63
net.(*pollDesc).WaitRead(0xc208010e60, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x51
net.(*netFD).Read(0xc208010e00, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x11bcb48, 0xc20800b160)
    /usr/local/go/src/net/fd_unix.go:242 +0x4b3
net.(*conn).Read(0xc208036090, 0xc20800f000, 0x1000, 0x1000, 0xc208010f38, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0x125
net.(*TCPConn).Read(0xc208036090, 0xc20800f000, 0x1000, 0x1000, 0xc20807bf40, 0x0, 0x0)
    <autogenerated>:36 +0x86
net/http.noteEOFReader.Read(0x11bdee0, 0xc208036090, 0xc208086058, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x7c
net/http.(*noteEOFReader).Read(0xc20801eac0, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:125 +0xfc
bufio.(*Reader).fill(0xc208052660)
    /usr/local/go/src/bufio/bufio.go:97 +0x382
bufio.(*Reader).Peek(0xc208052660, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0x199
net/http.(*persistConn).readLoop(0xc208086000)
    /usr/local/go/src/net/http/transport.go:842 +0xff
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0x10d4

goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc208086000)
    /usr/local/go/src/net/http/transport.go:945 +0x4ff
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0x10f1

goroutine 12 [select]:
net/http.(*persistConn).writeLoop(0xc2080860b0)
    /usr/local/go/src/net/http/transport.go:945 +0x4ff
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0x10f1
exit status 2

[olivere]

Hmm... sorry for the ping-pong, but now I'm curious. There is a nil pointer in your code that I don't have...

Which Go/OS version are you on? What's the output of go version and uname -a?

Also, if you're on Mac or Linux, can you compile the app with go build <code>.go and send me the binary so I can debug?

[emilgpa]

MAC OS X Yosemite 10.10

go version:

go version go1.4.2 darwin/amd64

uname -a:

Darwin Mac-Emilio 14.1.0 Darwin Kernel Version 14.1.0: Mon Dec 22 23:10:38 PST 2014; root:xnu-2782.10.72~2/RELEASE_X86_64 x86_64

Here you have the binary file ;)

https://www.dropbox.com/s/esjttgbp0qzsnzl/test_es.go?dl=0

[emilgpa]

Sorry, the link correct is:
https://www.dropbox.com/s/kr7to6g6fr6k9gd/test_es?dl=0

[olivere]

Just tested it in a VM for an hour. No connection problems here. I'm afraid it must have something to do with your network/environment.

Can you rewrite the program with an official client e.g. in Python or Ruby? For example, run gem i elasticsearch, then run this program with ruby <program>.rb:

#!/usr/bin/env ruby
# encoding: utf-8

require 'elasticsearch'

while true
  client = Elasticsearch::Client.new(log: true)
  client.transport.reload_connections!
  client.cluster.health
  client = nil
  sleep 1
end

[vsiv]

@olivere i see this error consistently when the network is down or when there isn't a functional network connection or even when there is a slow network connection (tether phone). wonder its in our usage that triggers this..

Was able to reproduce this at least twice - mac/yosemite/go1.4

[olivere]

@vishalshah-org Thanks. Hmm, tried to go into Wifi mode but can't reproduce it. I'm using Network Link Conditioner from Hardware IO Tools for Xcode now to simulate a flaky/slow network. But cannot get it to fail (yet).

What's interesting is that according to the stack trace it fails when performing a HTTP request. And it's crashing with a nil pointer, not even returning an error. I'm curious what the error might be, but I'm not sure I can do anything in Elastic to prevent this from happening.

That's why it'd be interesting if it also fails when using e.g. Ruby, as that might show that it's something related to Go, either Elastic or net/http.

[olivere]

@DasHaus Sorry for all the "spam" :-)

Besides the Ruby code, does this also yield the same problem?

package main

import (
    "fmt"
    "net/http"
    "time"
)

func main() {
    var n int64
    for {
        n++
        fmt.Printf("%d\r", n)

        res, err := http.DefaultClient.Get("http://127.0.0.1:9200/")
        if err != nil {
            fmt.Printf("%v\n", err)
        }
        if res.StatusCode != 200 {
            fmt.Printf("%s\n", res.StatusCode)
        }
        res.Body.Close()
        time.Sleep(100*time.Millisecond)
    }
}

[olivere]

Hmm... I could get it to fail with my example code. It happens if I connect to an IP that doesn't run ES. In my example code, there is a race in that I check an error but still continue. So the example runs on and then fails when doing the Ping. This is obviosuly a problem with my example code, so I have changed the gist. Does this still fail?

Also, I have pushed an issue-58 branch that enforces a check in NewClient, regardless if sniffing and/or healthchecks are enabled. I'm still investigating if there's another way to cause the problem.

[emilgpa]

@olivere I dont will be in my home to Monday. When i come back to home i will try it with your new instructions ;) . Thanks!

[vsiv]

@olivere i was outside and thats where i consistently saw the error with my tethered network connection. Now, from home I am not able to reproduce it too even with a tethered connection.

will continue testing...

[emilgpa]

@olivere The ruby code works fine. The net/http code works fine too. The new code of gist file still fails, but not by a panic (appeared to me twice the message "no Elasticsearch node available" but then continues normally).

[emilgpa]

@olivere The error occurs in the sniff method of Client struct, in the <-time.After(timeout) case of select statement.

I changed the value of DefaultSnifferTimeout const to 1 * (time.Second * 4) and the error is never raised.

[olivere]

Ha! That sounds like a very plausible explanation. Unfortunately, I won't
find the time in the next few days to check and fix it. It'll have to wait
for next weekend.

On 6 April 2015 at 23:07, dashaus [email protected] wrote:

@olivere https://github.com/olivere The error occurs in the sniff
method of Client struct, in the <-time.After(timeout) case of select
statement.

I changed the value of DefaultSnifferTimeout const to 1 * (time.Second *
4) and the error is never raised.

—
Reply to this email directly or view it on GitHub
#58 (comment).

[olivere]

Just for self-reference: The official Ruby client uses a timeout of 1s, the JS client uses 3s. The Python client uses .1s, but doesn't use that value when starting up the client. Docs:

:arg sniff_timeout: timeout used for the sniff request - it should be a
fast api call and we are talking potentially to more nodes so we want
to fail quickly. Not used during initial sniffing (if sniff_on_start is on)
when the connection still isn't initialized.

I lean to the Python way of doing things: Robust startup and relaxing/fast when running. That'd mean a combination of the changes by @DasHaus and the changes I made in branch issue-58.

[vsiv]

+1
that explains why it was so hard for me to repro from home..
kudos @DasHaus and @olivere

[olivere]

Guys, I've just pushed this commit in the hopes of fixing this issue.

Elastic now has configurable timeouts, both for health checks and the sniffing process. We have two types of timeouts: those used on startup (used in NewClient), and those used for subsequent processes. The former should help with slow/lossy networks. You can use elastic.SetSnifferTimeoutStartup(time.Duration) and elastic.SetHealthcheckTimeoutStartup(time.Duration) to change the (current) defaults of 5 seconds. If you also want to change the timeout for subsequent processes, you can do so with elastic.SetSnifferTimeout(time.Duration) and elastic.SetHealthcheckTimeout(time.Duration). These timeouts are more aggressive and (currently) default to 1 second.

Also, elastic.NewClient(...) now also ensures that we have at least one healthy connection to Elasticsearch before returning a client and no error.

Let me know if this still fails in your environment.

Thanks for helping to fix this, guys!

[nursejason]

Just wanted to add my comments to this issue. Today I ran into this exact issue while trying to connect to my elastic client. After two or three hours of messing around with different clients and setting timeouts that way, and using SetHealthcheckTimeoutStartup, I finally came here to see if anyone else had the same problem. My development environment is a local Vagrant image and my Elastic server is hosted remotely.

I never would have thought that the Sniffer timing out would have caused the error message mentioned in the original issue. I assumed a timeout was occurring, I just certainly didn't think it had something to do with the Sniffer. Perhaps it's due to my lack of experience! Though hopefully this will inspire a little more transparency in some way.

[olivere]

@nursejason So did the timeout settings solve the issue for you? Or is it still unresolved?

[nursejason]

I apologize for my lack of specificity!
SetHealthcheckTimeoutStartup did not resolve my issue, however the timeout setting SetSnifferTimeoutStartup did! It just wasn't abundantly clear to me through the docs or the error message that the Sniffer was causing my issue.

[olivere]

No need to apologize. Thanks.

Well, I reviewed the different timeouts in the official clients, but they don't seem to be standardized. Good defaults are hard to find, especially with different environments like local machines, slow/lossy networks, or containers in mind.

I'd hoped that the description of the client settings in the Wiki would be helpful, combined with the description of sniffing and health checks.

[nursejason]

Was looking at the Wiki when I had this, I just completely skipped over the sniffer. 5 second default will likely work for most people, guess I'm in a pretty edge case scenario. Regardless, just wanted to share my experience. Solid work!

[moomou]

I also ran into this issue when developing locally with ES v1.5.2 on OSX 10.10.4. Playing with the timeout like suggested in this thread didn't help.

Curiously, restarting my local ES makes this error go away.

Any clue on why this might be so is greatly appreciated.

Note: This consistently happens after it happens once, though I can't reliably reproduce it. Other clients I am using (nodejs) works fine.

[olivere]

@moo-mou Can you share more details please? What is the exact error you are seeing? Is ES on Docker or locally installed?

Here's what happens when you create a new Client: If sniffing is enabled, the Elastic will first start to sniff the cluster. If that fails, an error is returned. Otherwise it will make a healthcheck to find (at least) one healthy HTTP connection. Again, if that fails, no new client will be returned.

So if you don't get a new client with elastic.NewClient(...), one of those things must have failed. You can only play around with the Startup timeouts (both for the sniffer and the healthcheck).

Can you find out which one of the two processes fails for you? The exact error message should tell...

[moomou]

Hello @olivere, thanks for the prompt info.

I am running ES locally. No docker or intermediate proxy.

I see the failure message No Elastic node is available which I believe is coming from here

So far I haven't been able to reproduce it on demand. I will post again if I find a reliable way to reproduce it.

[olivere]

Maybe you can add some logging first to be sure where it's coming from.

There are two ErrNoClient errors with sniff. First when there is no healthy connection to any node in the cluster (all nodes seem to be dead). Second when none of the healthy connections responds in time (the default timeout is 5 seconds on startup but can be individually set with SetSnifferTimeoutStartup).

Maybe you can play around with that to reproduce.

[olivere]

@moo-mou Any news on this?

[moomou]

Sorry, didn't get any time to dig into this this weekend. I will take a look sometime this weekend.

[moomou]

Finally looked into this. tl;dr - my elastic server returns an invalid node IP with sniffing enabled so the go elastic library can't connect. I believe elastic go is behaving correctly.

The issue is that ELASTIC_HOST/_nodes api returns an invalid IP address that go elastic library could not connect. By invalid I mean the returned IP address is not localhost or local IP. If I disable sniffing, the elastic go is forced to use localhost:9200 then everything works as expected. This is the reason why I don't see this issue in node client I am using - it just uses localhost:9200 and by default there is no sniffing.

If I restart elastic server, the new node takes on a valid local IP so go elastic works with sniffing enabled too.

Just to be more specific, the exact error is coming from this line.

[olivere]

@moo-mou Thanks for reviewing this.

I've set up a page in the wiki which summarizes findings related to connection problems and how to debug/fix them. Will keep it up to date whenever we find something new.

[bmelton]

Is this still considered fixed? I can curl my Elasticsearch host. https://gist.github.com/bmelton/390585f02b7c3e1ac0c1

I created a quick Python script that should read from the version, and it works correctly:
https://gist.github.com/bmelton/b6de2b73dd8068bc8738

Output is

$ python connect.py 
1.7.3

But when I run the connect.go script from the gist you posted, all I get is

no Elasticsearch node available

I'm running go 1.5.1, on OSX, and I've tried using both "github.com/olivere/elastic" and "gopkg.in/olivere/elastic.v2" as dependencies, both of which yield the same thing. I've tried elastic.SetSniff(false), and I've tried setting the timeout to 10 seconds, but all of which give the same "no Elasticsearch node available" error.

The elasticsearch server is running on a Docker image on OSX put together with Docker compose, but even if I just visit the URL (http://192.168.99.100:9200) in my browser, I get a response back.

[bmelton]

That did it. Thanks so much for the response.

[stefaneg]

I would suggest making false the default for SetSniff. I doubt that local installation of ES is the default anymore...

[olivere]

If the official clients use false by default, I will use false as well. Last time I looked, they enabled sniffing by default.

[olivere]

Sniffing does not have anything to do with local vs. remote. It's about: Do you want me to dynamically watch the cluster and add new nodes as well as remove unavailable nodes? The problem almost always has been about being unable to connect to the nodes from the client-side.