Comments (41)
Hi @bmelton. It's always been a problem with IP addresses up until now, so I guess that it's the same with you. No worries, we'll get this to work for you as well :-)
If you don't use any special configurations, elastic will use the default IP (127.0.0.1:9200
) to sniff all nodes on the ES cluster. It does that by using the network info API, similar to the one you used in your connect.py
. It parses the IP addresses of all nodes ("http_address" : "inet[/172.17.0.31:9200]",
), periodically checks them for availability and adds new nodes as they come. The process is described in the Wiki. See also: Connection problems.
If you disable sniffing, you must set the IP addresses of all nodes manually by SetURL(...)
. Notice that elastic would not find new nodes in your cluster then.
So if you've got only one ES node in your cluster and it's available at 192.168.99.100:9200, this is supposed to work:
client, err := elastic.NewClient(
elastic.SetSniff(false),
elastic.SetURL("http://192.168.99.100:9200"),
)
Cheers.
from elastic.
@DasHaus I ran a program over night in the hopes of finding your problem. It simple connected to ES (which starts the sniffing process), then waits a few milliseconds, rinse and repeat. I'm afraid, I didn't have a single error in over 1 mio. connection attempts.
I'm also using ES locally on 127.0.0.1:9200 when developing and I didn't have that error once. So either I'm really lucky or there's something different between the way you are using ES/Elastic and I'm using it. Well, I'm at 1.5.0 and you're at 1.3.4, but that shouldn't be a problem. Go is 1.4.2 here.
Do you create one client for your app, or are you creating clients on demand?
from elastic.
I create one client for my app. For me, it's stranger, not always occurs... but when i do a change in my app, the server restarts and is there when, sometimes, occurs this error...
from elastic.
I can't get it to fail, even after 2 days. Can you run this gist for a while? If it fails, can you return the stack trace. Thank you.
from elastic.
Ok. I ran by go run <name_gist_file>
and within seconds, the following error is raised:
no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x9d701]
goroutine 1 [running]:
github.com/olivere/elastic.(*PingService).Do(0xc20803b1a0, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/ping.go:102 +0x471
github.com/olivere/elastic.(*Client).ElasticsearchVersion(0x0, 0x405290, 0x15, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:827 +0x9b
main.main()
/Users/emilio/Dropbox/test_es.go:28 +0x241
goroutine 5 [select]:
net/http.(*persistConn).roundTrip(0xc208086000, 0xc20800adb0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(*Transport).RoundTrip(0xc208038090, 0xc2080320d0, 0xc20803b020, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc2080320d0, 0x687df0, 0xc208038090, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(*Client).send(0x5b7be0, 0xc2080320d0, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(*Client).doFollowingRedirects(0x5b7be0, 0xc2080320d0, 0x483620, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(*Client).Do(0x5b7be0, 0xc2080320d0, 0xc20803afc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/olivere/elastic.(*Client).sniffNode(0xc208080000, 0xc20801e960, 0x15, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:543 +0x16a
github.com/olivere/elastic.func·014(0xc20801e960, 0x15)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x47
created by github.com/olivere/elastic.(*Client).sniff
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x744
goroutine 8 [IO wait]:
net.(*pollDesc).Wait(0xc208010df0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208010df0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208010d90, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x687b48, 0xc20800ae28)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc208036080, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x688fb8, 0xc208036080, 0xc208086058, 0xc20800f000, 0x1000, 0x1000, 0x676010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc20801ea80, 0xc20800f000, 0x1000, 0x1000, 0xc207ffc9f7, 0x0, 0x0)
<autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc20805a600)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc20805a600, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
exit status 2
from elastic.
Can you run with go run -race .go and send the output on failure, please?
Am 02.04.2015 um 20:13 schrieb dashaus [email protected]:
Ok. I ran by go run <name_gist_file> and within seconds, the following error is raised:
no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x9d701]
goroutine 1 [running]:
github.com/olivere/elastic.(_PingService).Do(0xc20803b1a0, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/ping.go:102 +0x471
github.com/olivere/elastic.(_Client).ElasticsearchVersion(0x0, 0x405290, 0x15, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:827 +0x9b
main.main()
/Users/emilio/Dropbox/test_es.go:28 +0x241
goroutine 5 [select]:
net/http.(_persistConn).roundTrip(0xc208086000, 0xc20800adb0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0x7ad
net/http.(_Transport).RoundTrip(0xc208038090, 0xc2080320d0, 0xc20803b020, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x558
net/http.send(0xc2080320d0, 0x687df0, 0xc208038090, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x4fc
net/http.(_Client).send(0x5b7be0, 0xc2080320d0, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x15b
net/http.(_Client).doFollowingRedirects(0x5b7be0, 0xc2080320d0, 0x483620, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xb25
net/http.(_Client).Do(0x5b7be0, 0xc2080320d0, 0xc20803afc0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xa4
github.com/olivere/elastic.(_Client).sniffNode(0xc208080000, 0xc20801e960, 0x15, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:543 +0x16a
github.com/olivere/elastic.func·014(0xc20801e960, 0x15)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x47
created by github.com/olivere/elastic.(*Client).sniff
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x744
goroutine 8 [IO wait]:
net.(_pollDesc).Wait(0xc208010df0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(_pollDesc).WaitRead(0xc208010df0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(_netFD).Read(0xc208010d90, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x687b48, 0xc20800ae28)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(_conn).Read(0xc208036080, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
net/http.noteEOFReader.Read(0x688fb8, 0xc208036080, 0xc208086058, 0xc20800f000, 0x1000, 0x1000, 0x676010, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(_noteEOFReader).Read(0xc20801ea80, 0xc20800f000, 0x1000, 0x1000, 0xc207ffc9f7, 0x0, 0x0)
:125 +0xd4
bufio.(_Reader).fill(0xc20805a600)
/usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(_Reader).Peek(0xc20805a600, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(_persistConn).readLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0xc9f
goroutine 9 [select]:
net/http.(_persistConn).writeLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(_Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0xcbc
exit status 2
—
Reply to this email directly or view it on GitHub.
from elastic.
no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0xc8da5]
goroutine 1 [running]:
github.com/olivere/elastic.(*PingService).Do(0xc20803acf0, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/ping.go:102 +0x565
github.com/olivere/elastic.(*Client).ElasticsearchVersion(0x0, 0x524eb0, 0x15, 0x0, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:827 +0xcf
main.main()
/Users/emilio/Dropbox/test_es.go:28 +0x2c0
goroutine 3822 [select]:
net/http.(*persistConn).roundTrip(0xc208086000, 0xc20800aad0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1082 +0xa69
net/http.(*Transport).RoundTrip(0xc208030000, 0xc2080324e0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:235 +0x6b8
net/http.send(0xc2080324e0, 0x11bcd18, 0xc208030000, 0xc20803ac00, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:219 +0x6f3
net/http.(*Client).send(0x725000, 0xc2080324e0, 0x21, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:142 +0x201
net/http.(*Client).doFollowingRedirects(0x725000, 0xc2080324e0, 0x5c06c0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:367 +0xf18
net/http.(*Client).Do(0x725000, 0xc2080324e0, 0xc20803ab40, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:174 +0xc4
github.com/olivere/elastic.(*Client).sniffNode(0xc2080d6000, 0xc2080f71c0, 0x15, 0x0, 0x0, 0x0)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:543 +0x19b
github.com/olivere/elastic.func·014(0xc2080f71c0, 0x15)
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x68
created by github.com/olivere/elastic.(*Client).sniff
/Users/emilio/go/src/github.com/olivere/elastic/client.go:508 +0x95f
goroutine 11 [IO wait]:
net.(*pollDesc).Wait(0xc208010370, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x63
net.(*pollDesc).WaitRead(0xc208010370, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x51
net.(*netFD).Read(0xc208010310, 0xc208089000, 0x1000, 0x1000, 0x0, 0x11bcb48, 0xc20800aea0)
/usr/local/go/src/net/fd_unix.go:242 +0x4b3
net.(*conn).Read(0xc208036068, 0xc208089000, 0x1000, 0x1000, 0x3a0e2, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0x125
net.(*TCPConn).Read(0xc208036068, 0xc208089000, 0x1000, 0x1000, 0x5c08d0, 0x0, 0x0)
<autogenerated>:36 +0x86
net/http.noteEOFReader.Read(0x11bdee0, 0xc208036068, 0xc208086108, 0xc208089000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x7c
net/http.(*noteEOFReader).Read(0xc20801e840, 0xc208089000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
<autogenerated>:125 +0xfc
bufio.(*Reader).fill(0xc2080526c0)
/usr/local/go/src/bufio/bufio.go:97 +0x382
bufio.(*Reader).Peek(0xc2080526c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0x199
net/http.(*persistConn).readLoop(0xc2080860b0)
/usr/local/go/src/net/http/transport.go:842 +0xff
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0x10d4
goroutine 8 [IO wait]:
net.(*pollDesc).Wait(0xc208010e60, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x63
net.(*pollDesc).WaitRead(0xc208010e60, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x51
net.(*netFD).Read(0xc208010e00, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x11bcb48, 0xc20800b160)
/usr/local/go/src/net/fd_unix.go:242 +0x4b3
net.(*conn).Read(0xc208036090, 0xc20800f000, 0x1000, 0x1000, 0xc208010f38, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0x125
net.(*TCPConn).Read(0xc208036090, 0xc20800f000, 0x1000, 0x1000, 0xc20807bf40, 0x0, 0x0)
<autogenerated>:36 +0x86
net/http.noteEOFReader.Read(0x11bdee0, 0xc208036090, 0xc208086058, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:1270 +0x7c
net/http.(*noteEOFReader).Read(0xc20801eac0, 0xc20800f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
<autogenerated>:125 +0xfc
bufio.(*Reader).fill(0xc208052660)
/usr/local/go/src/bufio/bufio.go:97 +0x382
bufio.(*Reader).Peek(0xc208052660, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:132 +0x199
net/http.(*persistConn).readLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:842 +0xff
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:660 +0x10d4
goroutine 9 [select]:
net/http.(*persistConn).writeLoop(0xc208086000)
/usr/local/go/src/net/http/transport.go:945 +0x4ff
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0x10f1
goroutine 12 [select]:
net/http.(*persistConn).writeLoop(0xc2080860b0)
/usr/local/go/src/net/http/transport.go:945 +0x4ff
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:661 +0x10f1
exit status 2
from elastic.
Hmm... sorry for the ping-pong, but now I'm curious. There is a nil pointer in your code that I don't have...
Which Go/OS version are you on? What's the output of go version
and uname -a
?
Also, if you're on Mac or Linux, can you compile the app with go build <code>.go
and send me the binary so I can debug?
from elastic.
MAC OS X Yosemite 10.10
go version:
go version go1.4.2 darwin/amd64
uname -a:
Darwin Mac-Emilio 14.1.0 Darwin Kernel Version 14.1.0: Mon Dec 22 23:10:38 PST 2014; root:xnu-2782.10.72~2/RELEASE_X86_64 x86_64
Here you have the binary file ;)
https://www.dropbox.com/s/esjttgbp0qzsnzl/test_es.go?dl=0
from elastic.
Sorry, the link correct is:
https://www.dropbox.com/s/kr7to6g6fr6k9gd/test_es?dl=0
from elastic.
Just tested it in a VM for an hour. No connection problems here. I'm afraid it must have something to do with your network/environment.
Can you rewrite the program with an official client e.g. in Python or Ruby? For example, run gem i elasticsearch
, then run this program with ruby <program>.rb
:
#!/usr/bin/env ruby
# encoding: utf-8
require 'elasticsearch'
while true
client = Elasticsearch::Client.new(log: true)
client.transport.reload_connections!
client.cluster.health
client = nil
sleep 1
end
from elastic.
@olivere i see this error consistently when the network is down or when there isn't a functional network connection or even when there is a slow network connection (tether phone). wonder its in our usage that triggers this..
Was able to reproduce this at least twice - mac/yosemite/go1.4
from elastic.
@vishalshah-org Thanks. Hmm, tried to go into Wifi mode but can't reproduce it. I'm using Network Link Conditioner from Hardware IO Tools for Xcode now to simulate a flaky/slow network. But cannot get it to fail (yet).
What's interesting is that according to the stack trace it fails when performing a HTTP request. And it's crashing with a nil pointer, not even returning an error. I'm curious what the error might be, but I'm not sure I can do anything in Elastic to prevent this from happening.
That's why it'd be interesting if it also fails when using e.g. Ruby, as that might show that it's something related to Go, either Elastic or net/http.
from elastic.
@DasHaus Sorry for all the "spam" :-)
Besides the Ruby code, does this also yield the same problem?
package main
import (
"fmt"
"net/http"
"time"
)
func main() {
var n int64
for {
n++
fmt.Printf("%d\r", n)
res, err := http.DefaultClient.Get("http://127.0.0.1:9200/")
if err != nil {
fmt.Printf("%v\n", err)
}
if res.StatusCode != 200 {
fmt.Printf("%s\n", res.StatusCode)
}
res.Body.Close()
time.Sleep(100*time.Millisecond)
}
}
from elastic.
Hmm... I could get it to fail with my example code. It happens if I connect to an IP that doesn't run ES. In my example code, there is a race in that I check an error but still continue. So the example runs on and then fails when doing the Ping
. This is obviosuly a problem with my example code, so I have changed the gist. Does this still fail?
Also, I have pushed an issue-58 branch that enforces a check in NewClient
, regardless if sniffing and/or healthchecks are enabled. I'm still investigating if there's another way to cause the problem.
from elastic.
@olivere I dont will be in my home to Monday. When i come back to home i will try it with your new instructions ;) . Thanks!
from elastic.
@olivere i was outside and thats where i consistently saw the error with my tethered network connection. Now, from home I am not able to reproduce it too even with a tethered connection.
will continue testing...
from elastic.
@olivere The ruby code works fine. The net/http code works fine too. The new code of gist file still fails, but not by a panic (appeared to me twice the message "no Elasticsearch node available" but then continues normally).
from elastic.
@olivere The error occurs in the sniff
method of Client
struct, in the <-time.After(timeout)
case of select
statement.
I changed the value of DefaultSnifferTimeout
const to 1 * (time.Second * 4)
and the error is never raised.
from elastic.
Ha! That sounds like a very plausible explanation. Unfortunately, I won't
find the time in the next few days to check and fix it. It'll have to wait
for next weekend.
On 6 April 2015 at 23:07, dashaus [email protected] wrote:
@olivere https://github.com/olivere The error occurs in the sniff
method of Client struct, in the <-time.After(timeout) case of select
statement.I changed the value of DefaultSnifferTimeout const to 1 * (time.Second *
4) and the error is never raised.—
Reply to this email directly or view it on GitHub
#58 (comment).
from elastic.
Just for self-reference: The official Ruby client uses a timeout of 1s, the JS client uses 3s. The Python client uses .1s, but doesn't use that value when starting up the client. Docs:
:arg sniff_timeout: timeout used for the sniff request - it should be a
fast api call and we are talking potentially to more nodes so we want
to fail quickly. Not used during initial sniffing (ifsniff_on_start
is on)
when the connection still isn't initialized.
I lean to the Python way of doing things: Robust startup and relaxing/fast when running. That'd mean a combination of the changes by @DasHaus and the changes I made in branch issue-58.
from elastic.
+1
that explains why it was so hard for me to repro from home..
kudos @DasHaus and @olivere
from elastic.
Guys, I've just pushed this commit in the hopes of fixing this issue.
Elastic now has configurable timeouts, both for health checks and the sniffing process. We have two types of timeouts: those used on startup (used in NewClient
), and those used for subsequent processes. The former should help with slow/lossy networks. You can use elastic.SetSnifferTimeoutStartup(time.Duration)
and elastic.SetHealthcheckTimeoutStartup(time.Duration)
to change the (current) defaults of 5 seconds. If you also want to change the timeout for subsequent processes, you can do so with elastic.SetSnifferTimeout(time.Duration)
and elastic.SetHealthcheckTimeout(time.Duration)
. These timeouts are more aggressive and (currently) default to 1 second.
Also, elastic.NewClient(...)
now also ensures that we have at least one healthy connection to Elasticsearch before returning a client and no error.
Let me know if this still fails in your environment.
Thanks for helping to fix this, guys!
from elastic.
Just wanted to add my comments to this issue. Today I ran into this exact issue while trying to connect to my elastic client. After two or three hours of messing around with different clients and setting timeouts that way, and using SetHealthcheckTimeoutStartup, I finally came here to see if anyone else had the same problem. My development environment is a local Vagrant image and my Elastic server is hosted remotely.
I never would have thought that the Sniffer timing out would have caused the error message mentioned in the original issue. I assumed a timeout was occurring, I just certainly didn't think it had something to do with the Sniffer. Perhaps it's due to my lack of experience! Though hopefully this will inspire a little more transparency in some way.
from elastic.
@nursejason So did the timeout settings solve the issue for you? Or is it still unresolved?
from elastic.
I apologize for my lack of specificity!
SetHealthcheckTimeoutStartup
did not resolve my issue, however the timeout setting SetSnifferTimeoutStartup
did! It just wasn't abundantly clear to me through the docs or the error message that the Sniffer was causing my issue.
from elastic.
No need to apologize. Thanks.
Well, I reviewed the different timeouts in the official clients, but they don't seem to be standardized. Good defaults are hard to find, especially with different environments like local machines, slow/lossy networks, or containers in mind.
I'd hoped that the description of the client settings in the Wiki would be helpful, combined with the description of sniffing and health checks.
from elastic.
Was looking at the Wiki when I had this, I just completely skipped over the sniffer. 5 second default will likely work for most people, guess I'm in a pretty edge case scenario. Regardless, just wanted to share my experience. Solid work!
from elastic.
I also ran into this issue when developing locally with ES v1.5.2 on OSX 10.10.4. Playing with the timeout like suggested in this thread didn't help.
Curiously, restarting my local ES makes this error go away.
Any clue on why this might be so is greatly appreciated.
Note: This consistently happens after it happens once, though I can't reliably reproduce it. Other clients I am using (nodejs) works fine.
from elastic.
@moo-mou Can you share more details please? What is the exact error you are seeing? Is ES on Docker or locally installed?
Here's what happens when you create a new Client: If sniffing is enabled, the Elastic will first start to sniff the cluster. If that fails, an error is returned. Otherwise it will make a healthcheck to find (at least) one healthy HTTP connection. Again, if that fails, no new client will be returned.
So if you don't get a new client with elastic.NewClient(...)
, one of those things must have failed. You can only play around with the Startup timeouts (both for the sniffer and the healthcheck).
Can you find out which one of the two processes fails for you? The exact error message should tell...
from elastic.
Hello @olivere, thanks for the prompt info.
I am running ES locally. No docker or intermediate proxy.
I see the failure message No Elastic node is available
which I believe is coming from here
So far I haven't been able to reproduce it on demand. I will post again if I find a reliable way to reproduce it.
from elastic.
Maybe you can add some logging first to be sure where it's coming from.
There are two ErrNoClient
errors with sniff
. First when there is no healthy connection to any node in the cluster (all nodes seem to be dead). Second when none of the healthy connections responds in time (the default timeout is 5 seconds on startup but can be individually set with SetSnifferTimeoutStartup
).
Maybe you can play around with that to reproduce.
from elastic.
@moo-mou Any news on this?
from elastic.
Sorry, didn't get any time to dig into this this weekend. I will take a look sometime this weekend.
from elastic.
Finally looked into this. tl;dr - my elastic server returns an invalid node IP with sniffing enabled so the go elastic library can't connect. I believe elastic go is behaving correctly.
The issue is that ELASTIC_HOST/_nodes api returns an invalid IP address that go elastic library could not connect. By invalid I mean the returned IP address is not localhost
or local IP. If I disable sniffing, the elastic go is forced to use localhost:9200 then everything works as expected. This is the reason why I don't see this issue in node client I am using - it just uses localhost:9200 and by default there is no sniffing.
If I restart elastic server, the new node takes on a valid local IP so go elastic works with sniffing enabled too.
Just to be more specific, the exact error is coming from this line.
from elastic.
@moo-mou Thanks for reviewing this.
I've set up a page in the wiki which summarizes findings related to connection problems and how to debug/fix them. Will keep it up to date whenever we find something new.
from elastic.
Is this still considered fixed? I can curl my Elasticsearch host. https://gist.github.com/bmelton/390585f02b7c3e1ac0c1
I created a quick Python script that should read from the version, and it works correctly:
https://gist.github.com/bmelton/b6de2b73dd8068bc8738
Output is
$ python connect.py
1.7.3
But when I run the connect.go script from the gist you posted, all I get is
no Elasticsearch node available
I'm running go 1.5.1, on OSX, and I've tried using both "github.com/olivere/elastic" and "gopkg.in/olivere/elastic.v2" as dependencies, both of which yield the same thing. I've tried elastic.SetSniff(false), and I've tried setting the timeout to 10 seconds, but all of which give the same "no Elasticsearch node available" error.
The elasticsearch server is running on a Docker image on OSX put together with Docker compose, but even if I just visit the URL (http://192.168.99.100:9200) in my browser, I get a response back.
from elastic.
That did it. Thanks so much for the response.
from elastic.
I would suggest making false the default for SetSniff. I doubt that local installation of ES is the default anymore...
from elastic.
If the official clients use false
by default, I will use false
as well. Last time I looked, they enabled sniffing by default.
from elastic.
Sniffing does not have anything to do with local vs. remote. It's about: Do you want me to dynamically watch the cluster and add new nodes as well as remove unavailable nodes? The problem almost always has been about being unable to connect to the nodes from the client-side.
from elastic.
Related Issues (20)
- I want to get the result body. Is there any way? HOT 1
- Generating nested histogram query HOT 1
- The es daterange query data has no expected effect
- how to get root cause with 400 error HOT 1
- ()elastic.Each Swallowed the error
- Unresolved reference 'Aggregation, Size, TrackTotalHits, Pretty, Sort, and Do'
- Append support max_concurrent_shard_requests in msearch
- Support for Version=0 in bulk_delete_request.go HOT 2
- The collapse query does not support multiple inner_hit conditions HOT 1
- elastic v6 not work correctly with http proxy
- elastic: Error 400 (Bad Request): only write ops with an op_type of create are allowed in data streams [type=illegal_argument_exception]
- Can't use REST compatibility headers
- The context field in the structure SearchSuggestionOption(in search.go) should change to be highlight
- How to search data with a 'knn query'?
- the return value of add document
- Issue with Elasticsearch Alias Removal and Addition
- cannot unmarshal object into Go struct field ProductHits.hits.total of type int
- EasyJson compatibility issue
- search without retry when es return io.EOF
- How can I use Offset in date_histogram of composite aggregation?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elastic.