Broken zookeeper build on AWS multinode about vagrant-mesos

[everpeace]

Firstly, thank you for your recent contributions!

chown: invalid user: 'zookeeper'
sudo: unknown user: zookeeper
sudo: unable to initialize policy plugin
/tmp/vagrant-shell: line 5: /etc/zookeeper/conf/zoo.cfg: No such file or directory
restart: Unknown job: zookeeper

This looks something went wrong. AMIs I published yesterday should have zookeeper service and 'zookeeper' user for sure.

Did you use custom_ami?? If you used custom_ami, you have to make sure that the box should have zookeeper installed.

[chrisjs]

I was using AMI ami-1a389d72 without any changes (from the http://cloud-images.ubuntu.com/locator/ec2/ site referenced in the old aws_region_ami.yaml) since i was using t2.medium instances and needed hvm support in us-east-1 / VPC. i was manually changing the aws_region_ami.yaml which lead me to the PR. My setup will still work if I keep my commits and prior, if i reset to the current master HEAD I get the above error.

So now I reset my master to the current head, modified my cluster config to NOT use the custom AMI and use the default instance types:
==> zk1: -- Type: m1.small
==> zk1: -- AMI: ami-9a50fff2

But after multiple attempts it just hangs continuously:
==> zk1: Waiting for instance to become "ready"...
==> zk1: Waiting for SSH to become available...

The instance is Running in AWS console, status checks claim all is well. Vagrant is hanging and I also can't manually ssh in. Everything stayed the same in my cluster config (i.e.: security groups etc) with the exception changing the instance type to m1.small and not specifying a custom AMI.

[chrisjs]

just as a sanity check below is a diff of the cluster.yml file that i was using before which was working fine and the mods i made not to use the custom AMI:

[multinodes]$ diff cluster.yml ~/cluster.yml 
52c52
< #custom_ami: ami-1a389d72
---
> custom_ami: ami-1a389d72
55,58c55,58
< zk_instance_type: m1.small
< master_instance_type: m1.small
< slave_instance_type: m1.small
< marathon_instance_type: m1.small
---
> zk_instance_type: t2.medium
> master_instance_type: t2.medium
> slave_instance_type: t2.medium
> marathon_instance_type: t2.medium

[chrisjs]

after re-reading your last comment and then looking at one of the commits about "pre-baked images" -- is it safe to say before things like zookeeper were being built when deploying instances and now with the "pre-baked" images it comes already installed?

so basically if i want to use a non-"pre-baked" AMI, id have to build an AMI with services such as zookeeper already installed? need to dig through the code a bit more but wanted to confirm my suspicion before getting to far in.

[everpeace]

@chrisjs sorry for your inconvenience.

now with the "pre-baked" images it comes already installed?

Yes, that's right. please refer to

• #19
• everpeace/mesos: the "pre-baked" Vagrant box used and shared on Vagrant Cloud
• everpeace/packer-mesos: Packer Templates which above box was baked.

so basically if i want to use a non-"pre-baked" AMI, id have to build an AMI
with services such as zookeeper already installed? need to dig through the code
a bit more but wanted to confirm my suspicion before getting to far in.

Basically, yes. "pre-baked" images is shipped with 4 projects bellow:

• Zookeeper
• Mesos
• Marathon
• Chronos

If you don't have such images, you can add "mesos" recipe and its configuration json like:

"mesos": {
  "type": "mesosphere",
    "version": "#{mesos_version}",
    "mesosphere": {
      "with_zookeeper": true
    }
}

If you need to install marathon and chronos, you can add installation scripts like this.

I will add some explanation about assumptions which the box used in Vagrantfile has.

[chrisjs]

ok thanks. running into a couple other issues as well -- for example when rebooting the marathon/chronos instance, neither of those services start again at boot, yet an unexpected zookeeper instance does (which didn't when the instance is first created). this is using the codebase prior to the pre-baked images. seems like either I'm using this in the wrong way or there may be other issues to work out.

going to close this issue for now as I'm not utilizing the new pre-baked ami.

thanks!

[huemans]

Apologies everpeace, Im trying to modify your configuration to run on DigitalOcean, and frankly I am a newbie. I dont have an image with Zookeeper etc so running into a similar challenge. This might be a horribly stupid question but I'd like to install Zookeeper via Salt/Chef as part of the config. Where would one add the above mesos JSON snippet in order to add that to the image/recipe?