GithubHelp home page GithubHelp logo

vivint-smarthome / ceph-on-mesos Goto Github PK

View Code? Open in Web Editor NEW
20.0 20.0 4.0 305 KB

Ceph on Mesos

Home Page: http://vivint-smarthome.github.io/ceph-on-mesos/

License: Apache License 2.0

Emacs Lisp 0.10% Scala 99.53% HTML 0.28% Makefile 0.03% Shell 0.06%
ceph framework mesos

ceph-on-mesos's People

Contributors

timcharper avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

mesosinfo fersantxez jacob-go marbelvega

ceph-on-mesos's Issues

Better documentation for multi-disk resource launching requirements for ceph

Using root / path resources for non-OSD daemons is fine.

Using root / path resources for OSD daemons is terrible and wrong and should be loudly advertised as such.

It would be considered ideal to have the framework allocate space up-front when root / path disk resources are used. Not sure if loopback-device mounts inside of a docker container are possible / sane.

Document how to do it manually.

OSD Mount Type Support?

It looks like only directory mode is used for osds according to this code block:
https://github.com/vivint-smarthome/ceph-on-mesos/blob/master/src/main/scala/com/vivint/ceph/JobBehavior.scala#L103

    val taskInfo = launchCephCommand(
      taskId = taskId,
      jobId = job.id,
      role = job.role,
      offer = offer,
      location = taskLocation,
      templatesTgz = templatesTgz,
      command =
        runState match {
          case RunState.Running =>
            s"""
            |FS_TYPE="$$(df -T /var/lib/ceph | tail -n 1 | awk '{print $$2}')"
            |if [ "$${FS_TYPE}" != "xfs" ] && [ "$${FS_TYPE}" != "btrfs" ]; then
            |  echo "Cowardly refusing to OSD start on non-xfs / non-btrfs volume."
            |  echo "Cowardly refusing to OSD start on non-xfs / non-btrfs volume." 1>&2
            |  echo "Please see http://docs.ceph.com/docs/jewel/rados/configuration/filesystem-recommendations/#not-recommended for more information"
            |  sleep 60
            |  exit
            |fi
            |set -x -e
            |echo "Pulling monitor map"
            |ceph mon getmap -o /etc/ceph/monmap-ceph
            |
            |if [ ! -f /etc/ceph/my_osd_id ]; then
            |  ceph osd create > /etc/ceph/my_osd_id
            |fi
            |OSD_ID=$$(cat /etc/ceph/my_osd_id)
            |mkdir -p /var/lib/ceph/osd/ceph-$${OSD_ID}
            |chown ceph:ceph /var/lib/ceph/osd/ceph-$${OSD_ID}
            |
            |exec /entrypoint.sh osd_directory
            |""".stripMargin
          case RunState.Paused =>
            s"""
            |sleep 86400
            |""".stripMargin
        }
    )
  1. Is there a way to change the directory used to be a volume mount like: /dcos/volume0
  2. Is there a way to use a disk mount like the ceph-docker project explained here:
    https://github.com/ceph/ceph-docker/tree/master/ceph-releases/jewel/ubuntu/14.04/daemon#ceph-disk

Implement uninstall endpoint

The framework should have a uninstall feature which kills all tasks and deallocates their resources.

original post

I couldn't find a way to cleanly uninstall the framework. I've also noticed that when I run this from DC/OS, if I uninstall by just killing the scheduler, ceph-mon and ceph-osd tasks are left orphaned.

Maybe related, I've also noticed that when launching the frameworks on DC/OS, ceph-mon and ceph-osd tasks appear in the Mesos UI as registered with the "ceph" framework. Nevertheless, in the DC/OS interface they don't appear inside the "ceph" Service, as the workers for other frameworks do.

A clarification on a clean way to uninstall the framework and cleaning all associated tasks would be much appreciated.

Better error message for zookeeper root node missing (when namespace specified in connection string)

Hi,
I was trying to deploy this framework, but finally get the errors as below, can you please provide some tips for the failure?

============= std error logs
22:54:39.115 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 15,3 replyHeader:: 15,55834603366,-101 request:: '/frameworks/ceph-on-mesos,F response::
22:54:39.170 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 16,1 replyHeader:: 16,55834603367,-101 request:: '/frameworks/ceph-on-mesos,,v{s{31,s{'world,'anyone}}},0 response::
22:54:45.237 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 17,1 replyHeader:: 17,55834603368,-101 request:: '/frameworks/ceph-on-mesos/master-lock/locks/_c_f707a2bb-25f0-42d6-ae36-24f2e13775a7-lock-,#3139322e3136382e3131312e3434,v{s{31,s{'world,'anyone}}},3 response::
22:54:45.241 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 18,3 replyHeader:: 18,55834603368,-101 request:: '/frameworks/ceph-on-mesos,F response::
22:54:45.295 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 19,1 replyHeader:: 19,55834603369,-101 request:: '/frameworks/ceph-on-mesos,,v{s{31,s{'world,'anyone}}},0 response::
22:54:45.298 [ceph-on-mesos-akka.actor.default-dispatcher-4-SendThread(192.168.111.55:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x25841ec9a19002e, packet:: clientPath:null serverPath:null finished:false header:: 20,4 replyHeader:: 20,55834603369,-101 request:: '/frameworks/ceph-on-mesos/ceph.conf,F response::
22:54:45.299 [ceph-on-mesos-akka.actor.default-dispatcher-8] ERROR com.vivint.ceph.TaskActor - acquiring lock: failure
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /ceph-on-mesos
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:291)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:741)
at org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:718)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:108)
at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:714)
at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.curator.framework.recipes.locks.StandardLockInternalsDriver.createsTheLock(StandardLockInternalsDriver.java:54)
at org.apache.curator.framework.recipes.locks.LockInternals.attemptLock(LockInternals.java:217)
at org.apache.curator.framework.recipes.locks.InterProcessMutex.internalLock(InterProcessMutex.java:232)
at org.apache.curator.framework.recipes.locks.InterProcessMutex.acquire(InterProcessMutex.java:89)
at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.internalAcquire1Lease(InterProcessSemaphoreV2.java:351)
at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.acquire(InterProcessSemaphoreV2.java:284)
at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreV2.acquire(InterProcessSemaphoreV2.java:212)
at org.apache.curator.framework.recipes.locks.InterProcessSemaphoreMutex.acquire(InterProcessSemaphoreMutex.java:46)
at com.vivint.ceph.kvstore.ZookeeperStore$$anonfun$lock$1.apply(ZookeeperStore.scala:117)
at com.vivint.ceph.kvstore.ZookeeperStore$$anonfun$lock$1.apply(ZookeeperStore.scala:114)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

TaskActor Fails to Start Up

Here is the section of the log I'm seeing the actor crash repeatedly. Let me know if you would like more context.

18:27:20.134 [ceph-on-mesos-akka.actor.default-dispatcher-3] DEBUG com.vivint.ceph.FrameworkActor - received 4 offers from mesos. Forwarding to TaskActor
18:27:33.399 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(10.158.15.164:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x158e0a1510b299a after 10ms
18:27:34.999 [ceph-on-mesos-akka.actor.default-dispatcher-7] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:130)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:36)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
18:27:35.012 [ceph-on-mesos-akka.actor.default-dispatcher-7] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-1/flow-0-0-unknown-operation#692057597]] terminated abruptly
18:27:35.037 [ceph-on-mesos-akka.actor.default-dispatcher-5] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
18:27:35.037 [ceph-on-mesos-akka.actor.default-dispatcher-5] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
18:27:35.040 [ceph-on-mesos-akka.actor.default-dispatcher-7] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
18:27:35.041 [ceph-on-mesos-akka.actor.default-dispatcher-2] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-4/flow-3-0-unknown-operation#-245605240]] terminated abruptly
18:27:35.045 [ceph-on-mesos-akka.actor.default-dispatcher-9] INFO  akka.actor.LocalActorRef - Message [akka.actor.Kill$] from Actor[akka://ceph-on-mesos/deadLetters] to Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor#-661139218] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
18:27:36.195 [ceph-on-mesos-akka.actor.default-dispatcher-10] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
18:27:36.196 [ceph-on-mesos-akka.actor.default-dispatcher-10] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state```

Support Kraken Release

config

docker_image = "ceph/daemon:tag-build-master-kraken-centos-7"

ceph -s

 cluster 0abeb1f9-4858-47c3-9da1-d5be421a85e7
     health HEALTH_WARN
            64 pgs degraded
            64 pgs stuck degraded
            64 pgs stuck unclean
            64 pgs stuck undersized
            64 pgs undersized
            too few PGs per OSD (21 < min 30)
     monmap e4: 3 mons at {10.0.0.10=10.0.0.10:6789/0,10.0.0.12=10.0.0.12:6789/0,10.0.0.5=10.0.0.5:6789/0}
            election epoch 8, quorum 0,1,2 10.0.0.5,10.0.0.10,10.0.0.12
        mgr no daemons active 
     osdmap e8: 3 osds: 3 up, 3 in
            flags sortbitwise,require_jewel_osds,require_kraken_osds
      pgmap v13: 64 pgs, 1 pools, 0 bytes data, 0 objects
            397 MB used, 3067 GB / 3067 GB avail
                  64 active+undersized+degraded

ceph mon getmap -o /etc/ceph/monmap-ceph:

got monmap epoch 4

ceph osd

 WEIGHT  TYPE NAME          UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 3.00000 root default                                         
-2       0     host 10.0.0.5                                    
-3       0     host 10.0.0.12                                   
-4       0     host 10.0.0.10                                   
-5 3.00000     host 10                                          
 2 1.00000         osd.2           up  1.00000          1.00000 
 0 1.00000         osd.0           up  1.00000          1.00000 
 1 1.00000         osd.1           up  1.00000          1.00000

Framework gets stuck in crash loop

Ceph on mesos got in a crash loop from which it couldn't recover. Here's the logs:

java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:08.941 [ceph-on-mesos-akka.actor.default-dispatcher-4527] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-187/flow-2178-0-unknown-operation#1053547476]] terminated abruptly
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4521] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:08.943 [ceph-on-mesos-akka.actor.default-dispatcher-4521] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-188/flow-2180-0-unknown-operation#-1470175774]] terminated abruptly
22:27:10.033 [ceph-on-mesos-akka.actor.default-dispatcher-4527] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:10.033 [ceph-on-mesos-akka.actor.default-dispatcher-4527] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:11.139 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:13.870 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59393"

22:27:13.871 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59393"

22:27:14.903 [ceph-on-mesos-akka.actor.default-dispatcher-4521] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:15.871 [ceph-on-mesos-akka.actor.default-dispatcher-4472] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59394"

22:27:15.871 [ceph-on-mesos-akka.actor.default-dispatcher-4472] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59394"

22:27:15.905 [ceph-on-mesos-akka.actor.default-dispatcher-4472] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:16.880 [ceph-on-mesos-akka.actor.default-dispatcher-4509] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59395"

22:27:16.880 [ceph-on-mesos-akka.actor.default-dispatcher-4509] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59395"

22:27:16.907 [ceph-on-mesos-akka.actor.default-dispatcher-4509] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:24.481 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:25.051 [ceph-on-mesos-akka.actor.default-dispatcher-4480] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:25.051 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-189/flow-2182-0-unknown-operation#116359274]] terminated abruptly
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4508] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:25.053 [ceph-on-mesos-akka.actor.default-dispatcher-4480] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-190/flow-2185-0-unknown-operation#590294963]] terminated abruptly
22:27:26.232 [ceph-on-mesos-akka.actor.default-dispatcher-4527] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:26.232 [ceph-on-mesos-akka.actor.default-dispatcher-4527] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:33.901 [ceph-on-mesos-akka.actor.default-dispatcher-4530] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59397"

22:27:33.901 [ceph-on-mesos-akka.actor.default-dispatcher-4530] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59397"

22:27:33.931 [ceph-on-mesos-akka.actor.default-dispatcher-4530] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:37.828 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:41.251 [ceph-on-mesos-akka.actor.default-dispatcher-4531] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:41.251 [ceph-on-mesos-akka.actor.default-dispatcher-4531] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-191/flow-2187-0-unknown-operation#-612276060]] terminated abruptly
22:27:41.253 [ceph-on-mesos-akka.actor.default-dispatcher-4532] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:41.253 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:41.253 [ceph-on-mesos-akka.actor.default-dispatcher-4508] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:41.254 [ceph-on-mesos-akka.actor.default-dispatcher-4527] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-192/flow-2189-0-unknown-operation#-1953777267]] terminated abruptly
22:27:42.312 [ceph-on-mesos-akka.actor.default-dispatcher-4535] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:42.312 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:44.920 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59399"

22:27:44.920 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59399"

22:27:44.947 [ceph-on-mesos-akka.actor.default-dispatcher-4534] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:45.920 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59400"

22:27:45.920 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59400"

22:27:45.948 [ceph-on-mesos-akka.actor.default-dispatcher-4534] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:46.920 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59401"

22:27:46.920 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59401"

22:27:47.951 [ceph-on-mesos-akka.actor.default-dispatcher-4508] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:27:51.161 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:27:57.330 [ceph-on-mesos-akka.actor.default-dispatcher-4535] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:27:57.330 [ceph-on-mesos-akka.actor.default-dispatcher-4535] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-193/flow-2191-0-unknown-operation#-260710826]] terminated abruptly
22:27:57.333 [ceph-on-mesos-akka.actor.default-dispatcher-4535] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:57.333 [ceph-on-mesos-akka.actor.default-dispatcher-4535] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:27:57.333 [ceph-on-mesos-akka.actor.default-dispatcher-4501] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:27:58.503 [ceph-on-mesos-akka.actor.default-dispatcher-4501] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:27:58.503 [ceph-on-mesos-akka.actor.default-dispatcher-4501] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:03.950 [ceph-on-mesos-akka.actor.default-dispatcher-4536] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59402"

22:28:03.950 [ceph-on-mesos-akka.actor.default-dispatcher-4536] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59402"

22:28:03.972 [ceph-on-mesos-akka.actor.default-dispatcher-4536] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:04.498 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:28:13.521 [ceph-on-mesos-akka.actor.default-dispatcher-4533] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:28:13.521 [ceph-on-mesos-akka.actor.default-dispatcher-4533] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-195/flow-2195-0-unknown-operation#1011439711]] terminated abruptly
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4533] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4531] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:28:13.526 [ceph-on-mesos-akka.actor.default-dispatcher-4536] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-196/flow-2198-0-unknown-operation#-1031974059]] terminated abruptly
22:28:14.673 [ceph-on-mesos-akka.actor.default-dispatcher-4538] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:14.673 [ceph-on-mesos-akka.actor.default-dispatcher-4538] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:14.961 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59403"

22:28:14.961 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59403"

22:28:14.989 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:15.960 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59404"

22:28:15.960 [ceph-on-mesos-akka.actor.default-dispatcher-4532] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59404"

22:28:15.991 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:17.832 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms
22:28:17.971 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Timing out offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59406"

22:28:17.971 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - Decline offer value: "12a792c8-8a5f-40b6-a4e3-56ea532aec76-O59406"

22:28:17.993 [ceph-on-mesos-akka.actor.default-dispatcher-4533] DEBUG com.vivint.ceph.FrameworkActor - received 1 offers from mesos. Forwarding to TaskActor
22:28:29.690 [ceph-on-mesos-akka.actor.default-dispatcher-4507] ERROR akka.actor.OneForOneStrategy - Timeout while initializing TaskActor
java.util.concurrent.TimeoutException: Timeout while initializing TaskActor
	at com.vivint.ceph.TaskActor$$anonfun$startInitialization$1.applyOrElse(TaskActor.scala:131)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
	at com.vivint.ceph.TaskActor.aroundReceive(TaskActor.scala:37)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
22:28:29.691 [ceph-on-mesos-akka.actor.default-dispatcher-4507] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-197/flow-2200-0-unknown-operation#-1749189163]] terminated abruptly
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4538] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4538] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4480] ERROR akka.actor.OneForOneStrategy - Kill
akka.actor.ActorKilledException: Kill
22:28:29.693 [ceph-on-mesos-akka.actor.default-dispatcher-4538] ERROR com.vivint.ceph.TaskActor - Unexpected error for configuration stream
akka.stream.AbruptTerminationException: Processor actor [Actor[akka://ceph-on-mesos/user/task-actor-backoff/task-actor/StreamSupervisor-198/flow-2202-0-unknown-operation#-1071175607]] terminated abruptly
22:28:30.743 [ceph-on-mesos-akka.actor.default-dispatcher-4539] INFO  com.vivint.ceph.TaskActor - pulling initial state for TaskActor
22:28:30.743 [ceph-on-mesos-akka.actor.default-dispatcher-4539] DEBUG com.vivint.ceph.TaskActor - acquiring lock : pulling state
22:28:31.179 [ceph-on-mesos-akka.actor.default-dispatcher-5-SendThread(172.31.29.134:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x15b5311724c00c4 after 0ms

Unsure of the reason. Killing the framework resolved it. Potential solution is to suicide after 10 failed retries?

Host Constraints

Can you impliment host constraints as only a few hosts in my cluster have xfs formatted drives.

Unresolved Dependencies

When building, I am running into the following issue.

Scala Version = 2.12.3
SBT Version = 1.0.0
Java Version 1.8

➜  ceph-on-mesos git:(master) ✗ sbt
[info] Loading settings from plugins.sbt ...
[info] Loading project definition from /ceph-on-mesos/project
[info] Updating {file:/ceph-on-mesos/project/}ceph-on-mesos-build...
[warn]  module not found: io.spray#sbt-revolver;0.8.0
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.typesafe.com/typesafe/ivy-releases/io.spray/sbt-revolver/scala_2.12/sbt_1.0//0.8.0/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/io.spray/sbt-revolver/scala_2.12/sbt_1.0//0.8.0/ivys/ivy.xml
[warn] ==== local: tried
[warn]   /root/.ivy2/local/io.spray/sbt-revolver/scala_2.12/sbt_1.0//0.8.0/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/io/spray/sbt-revolver_2.12_1.0/0.8.0/sbt-revolver-0.8.0.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /root/.sbt/preloaded/io.spray/sbt-revolver/0.8.0/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////root/.sbt/preloaded/io/spray/sbt-revolver_2.12_1.0/0.8.0/sbt-revolver-0.8.0.pom
[warn]  module not found: com.typesafe.sbt#sbt-native-packager;1.1.4
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.typesafe.com/typesafe/ivy-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0//1.1.4/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0//1.1.4/ivys/ivy.xml
[warn] ==== local: tried
[warn]   /root/.ivy2/local/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0//1.1.4/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/com/typesafe/sbt/sbt-native-packager_2.12_1.0/1.1.4/sbt-native-packager-1.1.4.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /root/.sbt/preloaded/com.typesafe.sbt/sbt-native-packager/1.1.4/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////root/.sbt/preloaded/com/typesafe/sbt/sbt-native-packager_2.12_1.0/1.1.4/sbt-native-packager-1.1.4.pom
[warn]  module not found: org.scala-js#sbt-scalajs;0.6.13
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.typesafe.com/typesafe/ivy-releases/org.scala-js/sbt-scalajs/scala_2.12/sbt_1.0//0.6.13/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.scala-js/sbt-scalajs/scala_2.12/sbt_1.0//0.6.13/ivys/ivy.xml
[warn] ==== local: tried
[warn]   /root/.ivy2/local/org.scala-js/sbt-scalajs/scala_2.12/sbt_1.0//0.6.13/ivys/ivy.xml
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/scala-js/sbt-scalajs_2.12_1.0/0.6.13/sbt-scalajs-0.6.13.pom
[warn] ==== local-preloaded-ivy: tried
[warn]   /root/.sbt/preloaded/org.scala-js/sbt-scalajs/0.6.13/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////root/.sbt/preloaded/org/scala-js/sbt-scalajs_2.12_1.0/0.6.13/sbt-scalajs-0.6.13.pom
[info] downloading https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.8.1/log4j-core-2.8.1-tests.jar ...
[info]  [SUCCESSFUL ] org.apache.logging.log4j#log4j-core;2.8.1!log4j-core.jar(test-jar) (419ms)
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: io.spray#sbt-revolver;0.8.0: not found
[warn]  :: com.typesafe.sbt#sbt-native-packager;1.1.4: not found
[warn]  :: org.scala-js#sbt-scalajs;0.6.13: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  Note: Some unresolved dependencies have extra attributes.  Check that these dependencies exist with the requested attributes.                                                  [4/3152]
[warn]          io.spray:sbt-revolver:0.8.0 (scalaVersion=2.12, sbtVersion=1.0)
[warn]          com.typesafe.sbt:sbt-native-packager:1.1.4 (scalaVersion=2.12, sbtVersion=1.0)
[warn]          org.scala-js:sbt-scalajs:0.6.13 (scalaVersion=2.12, sbtVersion=1.0)
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]          io.spray:sbt-revolver:0.8.0 (scalaVersion=2.12, sbtVersion=1.0) (/ceph-on-mesos/project/plugins.sbt#L1-2)
[warn]            +- default:ceph-on-mesos-build:0.1-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[warn]          com.typesafe.sbt:sbt-native-packager:1.1.4 (scalaVersion=2.12, sbtVersion=1.0) (/ceph-on-mesos/project/plugins.sbt#L3-4)
[warn]            +- default:ceph-on-mesos-build:0.1-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[warn]          org.scala-js:sbt-scalajs:0.6.13 (scalaVersion=2.12, sbtVersion=1.0) (/ceph-on-mesos/project/plugins.sbt#L5-6)
[warn]            +- default:ceph-on-mesos-build:0.1-SNAPSHOT (scalaVersion=2.12, sbtVersion=1.0)
[error] sbt.librarymanagement.ResolveException: unresolved dependency: io.spray#sbt-revolver;0.8.0: not found
[error] unresolved dependency: com.typesafe.sbt#sbt-native-packager;1.1.4: not found
[error] unresolved dependency: org.scala-js#sbt-scalajs;0.6.13: not found
[error]         at sbt.internal.librarymanagement.IvyActions$.resolveAndRetrieve(IvyActions.scala:331)
[error]         at sbt.internal.librarymanagement.IvyActions$.$anonfun$updateEither$1(IvyActions.scala:205)
[error]         at sbt.internal.librarymanagement.IvySbt$Module.$anonfun$withModule$1(Ivy.scala:229)
[error]         at sbt.internal.librarymanagement.IvySbt.$anonfun$withIvy$1(Ivy.scala:190)
[error]         at sbt.internal.librarymanagement.IvySbt.sbt$internal$librarymanagement$IvySbt$$action$1(Ivy.scala:70)
[error]         at sbt.internal.librarymanagement.IvySbt$$anon$3.call(Ivy.scala:77)
[error]         at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
[error]         at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
[error]         at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
[error]         at xsbt.boot.Using$.withResource(Using.scala:10)
[error]         at xsbt.boot.Using$.apply(Using.scala:9)
[error]         at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
[error]         at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
[error]         at xsbt.boot.Locks$.apply0(Locks.scala:31)
[error]         at xsbt.boot.Locks$.apply(Locks.scala:28)
[error]         at sbt.internal.librarymanagement.IvySbt.withDefaultLogger(Ivy.scala:77)
[error]         at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:185)
[error]         at sbt.internal.librarymanagement.IvySbt.withIvy(Ivy.scala:182)
[error]         at sbt.internal.librarymanagement.IvySbt$Module.withModule(Ivy.scala:228)
[error]         at sbt.internal.librarymanagement.IvyActions$.updateEither(IvyActions.scala:190)
[error]         at sbt.librarymanagement.ivy.IvyDependencyResolution.update(IvyDependencyResolution.scala:20)
[error]         at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:56)
[error]         at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:38)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:91)
[error]         at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:64)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$19(LibraryManagement.scala:104)
[error]         at scala.util.control.Exception$Catch.apply(Exception.scala:224)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:104)
[error]         at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:87)
[error]         at sbt.util.Tracked$.$anonfun$inputChanged$1(Tracked.scala:141)
[error]         at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:118)
[error]         at sbt.Classpaths$.$anonfun$updateTask$5(Defaults.scala:2313)
[error]         at scala.Function1.$anonfun$compose$1(Function1.scala:44)
[error]         at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:42)
[error]         at sbt.std.Transform$$anon$4.work(System.scala:64)
[error]         at sbt.Execute.$anonfun$submit$2(Execute.scala:257)
[error]         at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
[error]         at sbt.Execute.work(Execute.scala:266)
[error]         at sbt.Execute.$anonfun$submit$1(Execute.scala:257)
[error]         at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:167)
[error]         at sbt.CompletionService$$anon$2.call(CompletionService.scala:32)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? [error] (*:update) sbt.librarymanagement.ResolveException: unresolved dependency: io.spray#sbt-revolver;0.8.0: not found
[error] unresolved dependency: com.typesafe.sbt#sbt-native-packager;1.1.4: not found
[error] unresolved dependency: org.scala-js#sbt-scalajs;0.6.13: not found

SinglePortMatcher only considers the first range of resources

Given an offer with resources as following:

16:48:40.261 [ceph-on-mesos-akka.actor.default-dispatcher-26] DEBUG com.vivint.ceph.TaskActor - received offer
id {
  value: "c518e41e-8c8e-477f-b5eb-8610fe0c06fb-O82"
}
framework_id {
  value: "c518e41e-8c8e-477f-b5eb-8610fe0c06fb-0002"
}
slave_id {
  value: "c518e41e-8c8e-477f-b5eb-8610fe0c06fb-S2"
}
hostname: "172.31.11.151"
resources {
  name: "ports"
  type: RANGES
  ranges {
    range {
      begin: 1025
      end: 2180
    }
    range {
      begin: 2182
      end: 3887
    }
    range {
      begin: 3889
      end: 5049
    }
    range {
      begin: 5052
      end: 8079
    }
    range {
      begin: 8082
      end: 8180
    }
    range {
      begin: 8182
      end: 9667
    }
    range {
      begin: 9669
      end: 32000
    }
  }
  role: "*"
}

If the intention is to match port 6789, then it will not succeed. This is because the single port matcher for ceph-on-mesos is only looking at the first range.

Switch to leader election algorithm

Rather than acquiring lock in mesos, we should just have a single leader election instance that survives restarts of the task actor. Have taskActor wait for leadership, etc.

If we lose leadership, suicide, etc., etc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.