bakhti / ansible-elk Goto Github PK

They are not available yet upstream apparently, I wonder if there's a clean way for ansible to fall back on older distro repos such as trusty automatically, without spilling out and halting with this error.

TASK: [ansible-elk | Add nginx apt repo] ************************************** 
failed: [23.253.172.105] => {"failed": true, "parsed": false}
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1421313402.51-120126372595310/apt_repository", line 2466, in <module>
    main()
  File "/root/.ansible/tmp/ansible-tmp-1421313402.51-120126372595310/apt_repository", line 436, in main
    cache.update()
  File "/usr/lib/python2.7/dist-packages/apt/cache.py", line 440, in update
    raise FetchFailedException(e)
apt.cache.FetchFailedException: W:Failed to fetch http://nginx.org/packages/ubuntu/dists/utopic/nginx/binary-amd64/Packages  404  Not Found
, W:Failed to fetch http://nginx.org/packages/ubuntu/dists/utopic/nginx/binary-i386/Packages  404  Not Found
, E:Some index files failed to download. They have been ignored, or old ones used instead.


FATAL: all hosts have already failed -- aborting

@carles

This repo seems abandoned, but still.

Using the following ubuntu image in a VirtualBox Vagrantfile:

config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"

TASK: [elk | Modify config.js] ************************************************
failed: [default] => {"failed": true, "rc": 257}
msg: Destination /var/www/kibana/src/config.js does not exist !

FATAL: all hosts have already failed -- aborting

logstash 1.5 releases don't have logstash-contrib package anymore, which cause the install to fail.

vars:
  elk_logstash:
    version: 1.5

TASK: [bakhti.elk | Install logstash apt packages] ****************************
<xxx.xxx.xxx.xxx> REMOTE_MODULE apt pkg=logstash,logstash-contrib update_cache=yes state=present
failed: [xxx.xxx.xxx.xxx] => (item=logstash,logstash-contrib) => {"failed": true, "item": "logstash,logstash-contrib"}
msg: No package matching 'logstash-contrib' is available

Currently logstash and elasticsearch can be notified to restart after e.g. config change:

- name: Restart logstash
  service:
    name=logstash
    enabled=yes state=restarted
  sudo: yes

But that will actually spawn another logstash process in addition to the one monit handles. I think it needs to be:

- name: Restart logstash
  monit:
    name=logstash
    state=restarted
  sudo: yes

Keep up the good work,
Martin

• name: Install common apt packages
  apt: pkg={{ item }} state=latest
  with_items: elk_common_packages
  sudo: yes

Result:

failed: [172.32.1.82] => (item=openntpd,python-apt,python-pycurl,python-pip,supervisor,monit,git-core) => {"failed": true, "item": "openntpd,python-apt,python-pycurl,python-pip,supervisor,monit,git-core"}
stderr: /etc/openntpd/ntpd.conf: Permission denied
invoke-rc.d: initscript openntpd, action "start" failed.
dpkg: error processing package openntpd (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
openntpd
E: Sub-process /usr/bin/dpkg returned an error code (1)

stdout: Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
libopts25
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
git git-man liberror-perl python-meld3
Suggested packages:
git-daemon-run git-daemon-sysvinit git-doc git-el git-email git-gui gitk
gitweb git-arch git-bzr git-cvs git-mediawiki git-svn exim4 postfix
mail-transport-agent
The following packages will be REMOVED:
ntp
The following NEW packages will be installed:
git git-core git-man liberror-perl monit openntpd python-meld3 supervisor
0 upgraded, 8 newly installed, 1 to remove and 145 not upgraded.
Need to get 4021 kB of archives.
After this operation, 22.8 MB of additional disk space will be used.
Get:1 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/main liberror-perl all 0.17-1.1 [21.1 kB]
Get:2 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main git-man all 1:1.9.1-1ubuntu0.1 [698 kB]
Get:3 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main git amd64 1:1.9.1-1ubuntu0.1 [2627 kB]
Get:4 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty-updates/main git-core all 1:1.9.1-1ubuntu0.1 [1458 B]
Get:5 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe openntpd amd64 20080406p-7 [33.9 kB]
Get:6 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe monit amd64 1:5.6-2 [286 kB]
Get:7 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe python-meld3 amd64 0.6.10-1 [39.0 kB]
Get:8 http://us-east-1.ec2.archive.ubuntu.com/ubuntu/ trusty/universe supervisor all 3.0b2-1 [314 kB]
Fetched 4021 kB in 0s (11.6 MB/s)
(Reading database ... 56888 files and directories currently installed.)
Removing ntp (1:4.2.6.p5+dfsg-3ubuntu2.14.04.3) ...

• Stopping NTP server ntpd
  ...done.
  Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
  Selecting previously unselected package liberror-perl.
  (Reading database ... 56857 files and directories currently installed.)
  Preparing to unpack .../liberror-perl_0.17-1.1_all.deb ...
  Unpacking liberror-perl (0.17-1.1) ...
  Selecting previously unselected package git-man.
  Preparing to unpack .../git-man_1%3a1.9.1-1ubuntu0.1_all.deb ...
  Unpacking git-man (1:1.9.1-1ubuntu0.1) ...
  Selecting previously unselected package git.
  Preparing to unpack .../git_1%3a1.9.1-1ubuntu0.1_amd64.deb ...
  Unpacking git (1:1.9.1-1ubuntu0.1) ...
  Selecting previously unselected package git-core.
  Preparing to unpack .../git-core_1%3a1.9.1-1ubuntu0.1_all.deb ...
  Unpacking git-core (1:1.9.1-1ubuntu0.1) ...
  Selecting previously unselected package openntpd.
  Preparing to unpack .../openntpd_20080406p-7_amd64.deb ...
  Unpacking openntpd (20080406p-7) ...
  Selecting previously unselected package monit.
  Preparing to unpack .../monit_1%3a5.6-2_amd64.deb ...
  Unpacking monit (1:5.6-2) ...
  Selecting previously unselected package python-meld3.
  Preparing to unpack .../python-meld3_0.6.10-1_amd64.deb ...
  Unpacking python-meld3 (0.6.10-1) ...
  Selecting previously unselected package supervisor.
  Preparing to unpack .../supervisor_3.0b2-1_all.deb ...
  Unpacking supervisor (3.0b2-1) ...
  Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
  Processing triggers for ureadahead (0.100.0-16) ...
  Setting up liberror-perl (0.17-1.1) ...
  Setting up git-man (1:1.9.1-1ubuntu0.1) ...
  Setting up git (1:1.9.1-1ubuntu0.1) ...
  Setting up git-core (1:1.9.1-1ubuntu0.1) ...
  Setting up openntpd (20080406p-7) ...
  Starting openntpd: Setting up monit (1:5.6-2) ...
• Starting daemon monitor monit
  ...done.
  Setting up python-meld3 (0.6.10-1) ...
  Setting up supervisor (3.0b2-1) ...
  Starting supervisor: supervisord.
  Processing triggers for ureadahead (0.100.0-16) ...

msg: '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install 'openntpd' 'supervisor' 'monit' 'git-core'' failed: /etc/openntpd/ntpd.conf: Permission denied
invoke-rc.d: initscript openntpd, action "start" failed.
dpkg: error processing package openntpd (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
openntpd
E: Sub-process /usr/bin/dpkg returned an error code (1)

failed: [XXXXXX] => (item=openntpd,python-apt,python-pycurl,python-pip,supervisor,monit,git-core,jq) => {"failed": true, "item": "openntpd,python-apt,python-pycurl,python-pip,supervisor,monit,git-core,jq"}
msg: No package matching 'jq' is available

The json CLI is present only on Ubuntu
Is it mandatory ?

https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-repositories.html

Now it's packages.elastic.co instead of packages.elasticsearch.com, which 404s. Everything else appears to be the same.

Also, using the playbook with debian/openJDK, it breaks at the 'accept oracle license' step - need to change

 when: webupd8team.stdout != 'true'

to

"when: elk_java.flavor == 'oracle' and webupd8team.stdout != 'true' and ansible_distribution == 'Ubuntu'"

(better matches the previous steps)

Thanks for the playbook, by the way.

This role doesn't work with Ansible 2.
With default settings I'm getting the following and there is no way to override elk_java.flavor

TASK [elk : Java apt packages] *************************************************
failed: [localhost] => (item=[u'elk_java.openjdk']) => {"failed": true, "item": ["elk_java.openjdk"], "msg": "No package matching 'elk_java.openjdk' is available"}

Thanks for putting together a cool package.

I've had a crack at updating this, by just changing the version numbers to 2.1 but it seems to cause issues like this:

Fetched 250 kB in 4s (55.5 kB/s)
W: Failed to fetch http://packages.elastic.co/logstash/2.x/debian/dists/stable/main/binary-amd64/Packages  404  Not Found [IP: 23.21.242.59 80]

It looks like they have a debian/ubuntu repo for this now, does it make sense to use this instead? https://www.elastic.co/guide/en/logstash/current/package-repositories.html

Can you add some usage sample and a rundown of the variables with a single example?

The version of Elasticsearch which is embedded in logstash (currently 1.1.1) doesn't match with the one expected by Kibana (>1.4), making the latter fail. I workaround this by manually doing cp -R /usr/share/elasticsearch/ /opt/logstash/vendor/jar/elasticsearch-1.4.4 and then removing the old ES version in /opt/logstash/vendor/jar.
Maybe it's possible to add this task to the role so it keeps everything working? Do you think of a better approach for solving this? Let me know and I can work on a PR.

The change is substantial, now it runs on a dedicated script instead of having it deployed in a regular www server.