Comments (12)
satterly
commented on July 30, 2024
Can you login to the boxes and check for errors in any log files eg. /var/log/syslog ? When logged in also try to connect to the HTTP endpoint using curl. eg curl http://localhost:8080. Let me know what you find.
from elk-stack.
talbenbasat
commented on July 30, 2024
Hi,
This is the syslog from kibana instance:
c 21 16:02:20 ip-172-31-12-20 logcabin[27771]: events.js:141
Dec 21 16:02:20 ip-172-31-12-20 logcabin[27771]: throw er; // Unhandled 'error' event
Dec 21 16:02:20 ip-172-31-12-20 logcabin[27771]: ^
Dec 21 16:02:20 ip-172-31-12-20 logcabin[27771]: Error: connect ECONNREFUSED 127.0.0.1:9200
Dec 21 16:02:20 ip-172-31-12-20 logcabin[27771]: at Object.exports._errnoException (util.js:870:11)
Dec 21 16:02:20 ip-172-31-12-20 logcabin[27771]: at exports._exceptionWithHostPort (util.js:893:20)
Dec 21 16:02:20 ip-172-31-12-20 logcabin[27771]: at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1063:14)
Dec 21 16:02:20 ip-172-31-12-20 systemd[1]: logcabin.service: Main process exited, code=exited, status=1/FAILURE
Dec 21 16:02:20 ip-172-31-12-20 systemd[1]: logcabin.service: Unit entered failed state.
Dec 21 16:02:20 ip-172-31-12-20 systemd[1]: logcabin.service: Failed with result 'exit-code'.
Dec 21 16:02:20 ip-172-31-12-20 systemd[1]: logcabin.service: Service hold-off time over, scheduling restart.
Dec 21 16:02:20 ip-172-31-12-20 systemd[1]: Stopped logcabin.service.
Dec 21 16:02:20 ip-172-31-12-20 systemd[1]: Started logcabin.service.
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: Logcabin starting...
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: Google OAuth2 authentication used
Dec 21 16:02:21 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:21Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:21 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:21Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: Logcabin listening on 8080
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: events.js:141
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: throw er; // Unhandled 'error' event
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: ^
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: Error: connect ECONNREFUSED 127.0.0.1:9200
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: at Object.exports._errnoException (util.js:870:11)
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: at exports._exceptionWithHostPort (util.js:893:20)
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1063:14)
Dec 21 16:02:21 ip-172-31-12-20 systemd[1]: logcabin.service: Main process exited, code=exited, status=1/FAILURE
Dec 21 16:02:21 ip-172-31-12-20 systemd[1]: logcabin.service: Unit entered failed state.
Dec 21 16:02:21 ip-172-31-12-20 systemd[1]: logcabin.service: Failed with result 'exit-code'.
Dec 21 16:02:21 ip-172-31-12-20 systemd[1]: logcabin.service: Service hold-off time over, scheduling restart.
Dec 21 16:02:21 ip-172-31-12-20 systemd[1]: Stopped logcabin.service.
Dec 21 16:02:21 ip-172-31-12-20 systemd[1]: Started logcabin.service.
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27803]: Logcabin starting...
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27803]: Google OAuth2 authentication used
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27803]: Logcabin listening on 8080
Dec 21 16:02:22 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:22Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:22 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:22Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:23 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:23Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:23 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:23Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:24 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:24Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:24 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:24Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:26 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:26Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:26 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:26Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:27 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:27Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:27 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:27Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:28 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:28Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:28 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:28Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:29 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:29Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:29 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:29Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:31 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:31Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:31 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:31Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:32 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:32Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:32 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:32Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:32 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:32Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:32 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:32Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
Dec 21 16:02:33 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:33Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"Unable to revive connection: http://localhost:9200/"}
Dec 21 16:02:33 ip-172-31-12-20 kibana5[26908]: {"type":"log","@timestamp":"2017-12-21T16:02:33Z","tags":["warning","elasticsearch","admin"],"pid":26908,"message":"No living connections"}
its private ip is 172.31.12.20
This is the reponse from the curl:
Found. Redirecting to /auth/google
from elk-stack.
talbenbasat
commented on July 30, 2024
These are the instance permissions
Ports | Protocol | Source | elk-abc-ElkSecurityGroup-1AR94VO00LOT6
8080 | tcp | sg-xxxx1 | ✔
22 | tcp | 0.0.0.0/0 | ✔
6379 | tcp | sg-xxxx2 | ✔
9300-9400 | tcp | sg-xxxx3 | ✔
from elk-stack.
talbenbasat
commented on July 30, 2024
netstat
ubuntu@ip-172-31-12-20:~$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 200 ip-172-31-12-20.eu-:ssh static-37-142-40-:33168 ESTABLISHED
tcp6 0 0 ip-172-31-12-20.e:44064 52.119.240.0:https ESTABLISHED
tcp6 0 0 ip-172-31-12-2:http-alt ip-172-31-34-61.e:60588 TIME_WAIT
tcp6 0 0 ip-172-31-12-2:http-alt ip-172-31-34-61.e:60570 TIME_WAIT
tcp6 0 0 ip-172-31-12-2:http-alt ip-172-31-27-14.eu:6587 TIME_WAIT
tcp6 0 0 ip-172-31-12-2:http-alt ip-172-31-27-14.eu:6581 TIME_WAIT
tcp6 0 0 ip-172-31-12-2:http-alt ip-172-31-27-14.eu:6601 TIME_WAIT
tcp6 0 0 ip-172-31-12-2:http-alt ip-172-31-34-61.e:60578 TIME_WAIT
tcp6 1 0 localhost:53400 localhost:9200 CLOSE_WAIT
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
unix 3 [ ] DGRAM 826 /run/systemd/notify
unix 2 [ ] DGRAM 82431 /run/user/1000/systemd/notify
unix 2 [ ] DGRAM 836 /run/systemd/journal/syslog
unix 10 [ ] DGRAM 837 /run/systemd/journal/dev-log
unix 7 [ ] DGRAM 840 /run/systemd/journal/socket
unix 2 [ ] STREAM CONNECTED 14459
unix 2 [ ] STREAM CONNECTED 13402
unix 2 [ ] STREAM CONNECTED 13346
unix 3 [ ] STREAM CONNECTED 14968
unix 3 [ ] STREAM CONNECTED 101713 /run/systemd/journal/stdout
unix 2 [ ] DGRAM 14465
unix 2 [ ] DGRAM 14505
unix 2 [ ] STREAM CONNECTED 70757 /run/systemd/private
unix 2 [ ] STREAM CONNECTED 12152
unix 2 [ ] STREAM CONNECTED 44297
unix 3 [ ] STREAM CONNECTED 14473
unix 3 [ ] STREAM CONNECTED 14511 /var/run/dbus/system_bus_socket
unix 2 [ ] DGRAM 10489
unix 3 [ ] STREAM CONNECTED 102409
unix 2 [ ] STREAM CONNECTED 71083 /run/systemd/private
unix 2 [ ] STREAM CONNECTED 70371 /run/systemd/private
unix 3 [ ] STREAM CONNECTED 14508
unix 2 [ ] DGRAM 14496
unix 2 [ ] DGRAM 28127
unix 3 [ ] STREAM CONNECTED 14969 /var/run/dbus/system_bus_socket
unix 2 [ ] STREAM CONNECTED 15583
unix 2 [ ] STREAM CONNECTED 9841
unix 3 [ ] STREAM CONNECTED 14509
unix 2 [ ] STREAM CONNECTED 14568
unix 2 [ ] STREAM CONNECTED 13601
unix 2 [ ] DGRAM 14727
unix 3 [ ] DGRAM 11050
unix 2 [ ] STREAM CONNECTED 45769
unix 3 [ ] STREAM CONNECTED 81534
unix 2 [ ] DGRAM 11044
unix 3 [ ] DGRAM 11051
unix 3 [ ] DGRAM 11048
unix 3 [ ] DGRAM 9845
unix 3 [ ] DGRAM 9846
unix 2 [ ] DGRAM 81406
unix 3 [ ] DGRAM 11049
unix 3 [ ] STREAM CONNECTED 75537
unix 2 [ ] DGRAM 14723
unix 2 [ ] DGRAM 81481
unix 3 [ ] STREAM CONNECTED 81468
unix 2 [ ] DGRAM 14275
unix 3 [ ] STREAM CONNECTED 14840 /var/run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 81535
unix 2 [ ] DGRAM 81474
unix 3 [ ] STREAM CONNECTED 14510 /var/run/dbus/system_bus_socket
unix 2 [ ] STREAM CONNECTED 45524
unix 3 [ ] STREAM CONNECTED 13432
unix 3 [ ] STREAM CONNECTED 82401 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 14839
unix 2 [ ] STREAM CONNECTED 9435
unix 3 [ ] STREAM CONNECTED 73257 /run/systemd/journal/stdout
unix 2 [ ] DGRAM 72919
unix 2 [ ] STREAM CONNECTED 11029
unix 2 [ ] DGRAM 72948
from elk-stack.
talbenbasat
commented on July 30, 2024
Hi @satterly,
Any further ideas or guidance :)
from elk-stack.
talbenbasat
commented on July 30, 2024
Hi @satterly,
any further help
from elk-stack.
satterly
commented on July 30, 2024
the logcabin service is clearly not starting up correctly. You need to find out what is the root cause of this error...
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: Logcabin listening on 8080
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: events.js:141
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: throw er; // Unhandled 'error' event
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: ^
Dec 21 16:02:21 ip-172-31-12-20 logcabin[27792]: Error: connect ECONNREFUSED 127.0.0.1:920
from elk-stack.
talbenbasat
commented on July 30, 2024
@satterly how should it look like when its working?
Any suggestions where to start looking?
from elk-stack.
satterly
commented on July 30, 2024
Log in to the kibana host and run logcabin in the foreground then try to use kibana. You should get some better output. If you don't see any more logging edit the logcabin javascript file app.js and add your own console.log() statements (example https://github.com/guardian/elk-stack/blob/master/src/app.js#L20) until you can work out what's going on.
from elk-stack.
talbenbasat
commented on July 30, 2024
Hi,
Started to debug the logcabin.
it appears to be an Authentication problem.
Im failing in notAuthenticated function.
this is the req.url /__es/.
url.indexOf('/auth/google') returns -1
and config.oauth_unauthenticated.indexOf(url) returns 0.
this causes
config.oauth_unauthenticated.indexOf(url) > -1 to be true
from elk-stack.
talbenbasat
commented on July 30, 2024
Hi @satterly,
Fixed some issues. (google+ was disabled)
now the error from the syslog is gone. :)
but i still cant access to the kibana. :/
from elk-stack.
talbenbasat
commented on July 30, 2024
Working
google issue with the callback and wrong url
Thanks 🍾
from elk-stack.
Related Issues (20)
- AMI doesn't start ES? HOT 1
- Add support for new instance types HOT 2
- Security: cookie_secret is hard-coded HOT 1
- Template validation error undefined resource ElkLoadBalancer HOT 1
- ELKStackMultiAZinPrivateVPC HOT 1
- Special characters in CookieSecret parameter can break user data script HOT 1
- AWS::IAM::Policy - ElkS3Policy At least one of [Groups,Roles,Users] must be non-empty. HOT 1
- nodejs application crashes when accessing undefined property to match domains to config on google oauth. HOT 1
- Kibana 4 support? HOT 5
- Blank white screen: HOT 2
- Blank white screen HOT 3
- InternalOAuthError: failed to fetch user profile HOT 1
- Launch issue HOT 11
- Variables are not set to /opt/logcabin/config.js HOT 2
- Use hapi to implement the authentication proxy HOT 1
- Investigate what is required to upgrade to Elastic Stack 5.0 HOT 2
- eu-west-2 AMI ID? HOT 4
- How to add a log to the ELK
- Remove Google Auth
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elk-stack.