portworx / px-dev

PX-Developer is scale-out storage for containers. Run Cassandra, Jenkins, or any application in Docker, with enterprise storage functionality on commodity servers

PX-Developer

PX-Developer (PX-Dev) is scale-out storage and data services for containers. PX-Dev itself is deployed as a container with your application stack. By running PX-Dev with your application stack, you get container-granular controls for storage persistence, capacity management, performance, and availability in a scaleout environment. Deploying the PX-Developer container on a server with Docker Engine turns that server into a scale-out storage node. Storage runs converged with compute and gives bare-metal drive performance. PX-Dev can be used alongside Docker to provide persistent storage for containerized applications to get familiarized with running persistent storage use-cases for containerized applications. Portworx recommends using PX-Enterprise for production installations for all orchestrators including Kubernetes, DC/OS, Nomad, Docker EE and more.

PX-Dev offers container granular services such as:

1. Data persistence in a multi node environment
2. Synchronous data availability across multiple availability zones and automatic AZ detection
3. Automatic tiering and class of service enforcement
4. Bring-your-own-key Encryption
5. Shared namespaces across containers running on different servers
6. S3 interfaces and backup to S3
7. Integration with schedulers to automate container placement

Please visit our offical docs site for more information on running Portworx

Visit our website to learn more about some of the most common use cases:

Docker persistent storage Kubernetes storage DCOS persistent storage

Join us on slack @ http://slack.portworx.com/

px-dev's Issues

Shared Volume did not work

Regarding that information: http://docs.portworx.com/shared-volumes.html
Shared Volume is available within any Container on any Host.

However, thats not possible with the following setup:
https://github.com/Berndinox/swarm_portworx

OS: Ubuntu 16.04
Docker: 1.13 in Swarm Mode

Outputs:
Status:

Status: PX is operational
Node ID: 4d55c7f8-eda0-41ba-aec9-029aa8fd4ffe
        IP: 10.4.69.xx
        Local Storage Pool: 1 pool
        Pool    IO_Priority     Size    Used    Status  Zone    Region
        0       HIGH            47 GiB  2.1 GiB Online  default default
        Local Storage Devices: 1 device
        Device  Path            Media Type              Size            Last-Scan
        0:1     /dev/vdb        STORAGE_MEDIUM_MAGNETIC 47 GiB          07 Feb 17 07:59 UTC
        total                   -                       47 GiB
Cluster Summary
        Cluster ID: swarm
        Node IP: 10.4.55.xx - Capacity: 2.1 GiB/47 GiB Online
        Node IP: 10.4.69.xx - Capacity: 2.1 GiB/47 GiB Online (This node)
        Node IP: 10.4.65.xx - Capacity: 2.1 GiB/47 GiB Online
Global Storage Pool
        Total Used      :  6.4 GiB
        Total Capacity  :  140 GiB

volume inspect:

      Name                     :  data
       Size                     :  5.0 GiB
       Format                   :  ext4
       HA                       :  3
       IO Priority              :  LOW
       Shared                   :  yes
       Status                   :  up
       State                    :  Attached: b60f25de-09c7-414b-b886-fbb022f14b56
       Device Path              :  /dev/pxd/pxd572196087553633467
       Reads                    :  5
       Reads MS                 :  0
       Bytes Read               :  20480
       Writes                   :  1
       Writes MS                :  30
       Bytes Written            :  4096
       IOs in progress          :  0
       Bytes used               :  131 MiB
       Replica sets on nodes:
               Set  0
                       Node     :  10.4.55.xx
                       Node     :  10.4.65.xx
                       Node     :  10.4.69.xx

Swarm has 3 Members, i do start the 2 same container with the following on 2 different nodes:

docker run -it -v data:/data --volume-driver pxd tumtum/hello-world sh

There is the same behavior if i start a docker swarm service and replicate the containers to different hosts.

docker inspct shows:

        "Mounts": [
            {
                "Type": "volume",
                "Name": "data",
                "Source": "/var/lib/osd/mounts/data",
                "Destination": "/data",
                "Driver": "pxd",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],

However the volume just appear to be local to the node. Changes on Container 1 ond Node one are not visible on Container 2 on Node 2.

px-dev logs:

Server1:

time="2017-02-07 07:44:13Z" level=INFO msg="process_cdb_update: v: 21784 0 mounting device 572196087553633467 was on -1"
time="2017-02-07 07:44:13Z" level=INFO msg="ResyncRequest: dev 572196087553633467 v 21784 from [0 1 2] to [0 1 2]"
time="2017-02-07 07:44:13Z" level=INFO msg="update_cdb: dev: 572196087553633467 rset: 0 node[ 0 1 2 ] curr[ 0 1 2 ] next[ 0 1 2 ] new_rset [ empty ] remove [ empty ] pool_ids [ 0 0 0 ]  new_pool_ids [ empty ]"
time="2017-02-07T07:44:13Z" level=info msg="volumeStateHandler update" AbortOnError=false Driver=pxd Error=<nil> Format=FS_TYPE_EXT4 Function=d.volumePut ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21784
time="2017-02-07T07:44:13Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21791
time="2017-02-07T07:44:13Z" level=warning msg="No historical stats for device" AbortOnError=false Driver=pxd Error=<nil> Format=FS_TYPE_EXT4 Function=Attach ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21791
time="2017-02-07T07:44:13Z" level=info msg="Volume 572196087553633467:data exported at /var/lib/osd/pxns/572196087553633467"
time="2017-02-07T07:44:13Z" level=info msg="Attached namespace volume data"
time="2017-02-07T07:44:13Z" level=info msg="Mounting namespace volume 572196087553633467 (data) at /var/lib/osd/mounts/data"
Adding subvol data to namespace 572196087553633467
time="2017-02-07T07:44:13Z" level=info msg="Requesting namespace attachment for volume 572196087553633467"
time="2017-02-07T07:44:13Z" level=info msg="Volume 572196087553633467 is already exported"
time="2017-02-07T07:44:13Z" level=info msg="Attached namespace volume data"
time="2017-02-07T07:44:13Z" level=info msg="Namespace volume 572196087553633467 attached on 10.4.55.5"
time="2017-02-07T07:44:13Z" level=info msg="Locally attached namespace Vol"
time="2017-02-07T07:44:13Z" level=info msg="Volume 572196087553633467 is attached on 0x7fdbd0000ca0"
Created virtual path /pxmounts/572196087553633467
time="2017-02-07T07:44:13Z" level=info msg="Mounted namespace volume 572196087553633467 at /var/lib/osd/mounts/data"
time="2017-02-07T07:44:13Z" level=info msg="response /var/lib/osd/mounts/data" Driver=pxd ID=data Request=mount
time="2017-02-07T07:45:36Z" level=info Driver=pxd ID=572196087553633467 Request=volumeSet
time="2017-02-07T07:45:36Z" level=info msg="Will not detach volume 572196087553633467 because it is mounted"
time="2017-02-07T07:45:44Z" level=info Driver=pxd ID=572196087553633467 Request=volumeSet
time="2017-02-07T07:45:44Z" level=info msg="Volume 572196087553633467 is already exported"
time="2017-02-07T07:45:44Z" level=info msg="Attached namespace volume data"
time="2017-02-07T07:45:45Z" level=info Driver=pxd ID=572196087553633467 Request=volumeSet
time="2017-02-07T07:45:45Z" level=info msg="Volume 572196087553633467 is already exported"
time="2017-02-07T07:45:45Z" level=info msg="Attached namespace volume data"
time="2017-02-07T07:47:39Z" level=info msg=postVolumeUsage AbortOnError=false Driver=pxd Error=<nil> Format=FS_TYPE_EXT4 Function=d.volumePut ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21791
time="2017-02-07T07:47:39Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLU

Server2:

time="2017-02-07T07:39:24Z" level=info msg=Done Function=grpc-client.BlockCreate Tag=89731
time="2017-02-07T07:39:24Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21433
time="2017-02-07T07:39:24Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21440
time="2017-02-07T07:39:27Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21450
time="2017-02-07T07:39:27Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_DETATCHING Version=21451
time="2017-02-07T07:39:27Z" level=info msg="Action: 2 data <nil>" AttachedOn= Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_DETACHED Version=21457
time="2017-02-07T07:42:49Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21533
time="2017-02-07T07:42:49Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21545
time="2017-02-07T07:43:07Z" level=info msg="Mounting namespace volume 572196087553633467 (data) at /var/lib/osd/mounts/data"
Adding subvol data to namespace 572196087553633467
time="2017-02-07T07:43:07Z" level=info msg="Requesting namespace attachment for volume 572196087553633467"
time="2017-02-07T07:43:08Z" level=info msg="Namespace volume 572196087553633467 attached on 10.4.55.5"
time="2017-02-07T07:43:08Z" level=info msg="Volume 572196087553633467 is attached on 0x7ff7a0000f70"
Created virtual path /pxmounts/572196087553633467
time="2017-02-07T07:43:08Z" level=info msg="Mounted namespace volume 572196087553633467 at /var/lib/osd/mounts/data"
time="2017-02-07T07:43:08Z" level=info msg="response /var/lib/osd/mounts/data" Driver=pxd ID=data Request=mount
time="2017-02-07T07:43:32Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_DETATCHING Version=21657
time="2017-02-07T07:43:32Z" level=info msg="Action: 2 data <nil>" AttachedOn= Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_DETACHED Version=21663
time="2017-02-07T07:43:40Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21674
time="2017-02-07T07:43:40Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21685
time="2017-02-07T07:44:07Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_DETATCHING Version=21752
time="2017-02-07T07:44:07Z" level=info msg="Action: 2 data <nil>" AttachedOn= Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_DETACHED Version=21758
time="2017-02-07T07:44:13Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21784
time="2017-02-07T07:44:13Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21791
time="2017-02-07T07:45:36Z" level=info msg="Unmounted namespace volume 572196087553633467 at path /var/lib/osd/mounts/data"
time="2017-02-07T07:45:45Z" level=info msg="Mounting namespace volume 572196087553633467 (data) at /var/lib/osd/mounts/data"
Adding subvol data to namespace 572196087553633467
time="2017-02-07T07:45:45Z" level=info msg="Requesting namespace attachment for volume 572196087553633467"
time="2017-02-07T07:45:45Z" level=info msg="Namespace volume 572196087553633467 attached on 10.4.55.5"
time="2017-02-07T07:45:45Z" level=info msg="Volume 572196087553633467 is attached on 0x7ff7f0001200"
Created virtual path /pxmounts/572196087553633467
time="2017-02-07T07:45:45Z" level=info msg="Mounted namespace volume 572196087553633467 at /var/lib/osd/mounts/data"
time="2017-02-07T07:45:45Z" level=info msg="response /var/lib/osd/mounts/data" Driver=pxd ID=data Request=mount
time="2017-02-07T07:47:39Z" level=info msg="Action: 2 data <nil>" AttachedOn=b60f25de-09c7-414b-b886-fbb022f14b56 Driver=kernel Error=<nil> Function=VolumeStateChange ID=572196087553633467 State=VOLUME_STATE_ATTACHED Version=21957

non sense space usage

I have a cluster with 3 nodes,
all the volumes have replica 3
but if I check the used disk size for every host, it doesn't appear exactly the same size usage, should be if I understand correctly how works portworx

Cluster Summary Cluster ID: portworx Cluster UUID: ---- Nodes: 3 node(s) with storage (3 online) IP ID StorageNode Used Capacity Status 10.14.128.0 ---- Yes 70 GiB 200 GiB Online (This node) 10.10.16.0 ---- Yes 118 GiB 200 GiB Online 10.14.160.0 ---- Yes 71 GiB 200 GiB Online Global Storage Pool Total Used : 259 GiB Total Capacity : 600 GiB

Problems rebooting host running px-dev

As part of testing a px-dev POC, I have tried to reboot one of the hosts (RancherOS 0.9.0) that is part of my newly created px cluster.

Once the host has come up and all the containers have started, the node is not able to re-join the cluster. Something is preventing the startup of the px service.

After the startup attempts, I get a lot of the following in the log:

/03/2017 14:27:022017-03-28 12:27:02,793 INFO spawned: 'px_event_listener' with pid 4946
28/03/2017 14:27:032017-03-28 12:27:03,983 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:27:042017-03-28 12:27:04,208 INFO exited: px_event_listener (exit status 1; not expected)
28/03/2017 14:27:042017-03-28 12:27:04,208 ERRO pool px_event_listener event buffer overflowed, discarding event 463
28/03/2017 14:27:052017-03-28 12:27:05,281 INFO spawned: 'px_event_listener' with pid 4949
28/03/2017 14:27:062017-03-28 12:27:06,417 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:27:062017-03-28 12:27:06,625 INFO exited: px_event_listener (exit status 1; not expected)
28/03/2017 14:27:062017-03-28 12:27:06,625 ERRO pool px_event_listener event buffer overflowed, discarding event 464
28/03/2017 14:27:072017-03-28 12:27:07,710 INFO spawned: 'px_event_listener' with pid 4952

Restarting the container from Rancher (or from the Docker CLI), gives the following output:

9eea022bb016 9eea022bb016 0b3180a13460 cc3df29026cf 0720fcceaa4c d862d59d48b2 0bb26f652d6b 8f5add2719bb 134bcec2bdb9 9c736891bddd a11849ef8244 57567d8be17b  57567d8be17b b77a2a606112
2017-03-28 12:28:57,274 INFO stopped: pxdaemon (exit status 0)
28/03/2017 14:28:572017-03-28 12:28:57,277 INFO stopped: exec (exit status 2)
28/03/2017 14:28:57Waiting for data availability
28/03/2017 14:28:57Tracing stopped for session pxd
28/03/2017 14:28:57Session pxd destroyed
28/03/2017 14:28:57Exiting lttng
28/03/2017 14:28:572017-03-28 12:28:57,529 INFO stopped: lttng (exit status 0)
28/03/2017 14:28:572017-03-28 12:28:57,656 INFO stopped: relayd (exit status 0)
28/03/2017 14:29:04Tue Mar 28 12:29:04 UTC 2017 : Running on Linux nodeable-rancheros-pxd-003 4.9.15-rancher #1 SMP Wed Mar 15 05:07:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
28/03/2017 14:29:04size for /dev/shm is 67100672, less than required 293601280
28/03/2017 14:29:07PXD version:  ab65c7fb556554eeb279c49c3d3d571576422669
28/03/2017 14:29:08Using cluster: 14c12ce6-13a7-11e7-93ae-92361f002671
28/03/2017 14:29:08Key Value Store: etcd://10.42.226.249:2379
28/03/2017 14:29:08Using storage device: /dev/xvdc
28/03/2017 14:29:08/docker-entry-point.sh: line 721: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
28/03/2017 14:29:08Failed to enable rt scheduler
28/03/2017 14:29:102017-03-28 12:29:10,711 CRIT Supervisor running as root (no user in config file)
28/03/2017 14:29:102017-03-28 12:29:10,720 INFO supervisord started with pid 1
28/03/2017 14:29:112017-03-28 12:29:11,777 INFO spawned: 'relayd' with pid 67
28/03/2017 14:29:112017-03-28 12:29:11,802 INFO spawned: 'lttng' with pid 68
28/03/2017 14:29:112017-03-28 12:29:11,896 INFO spawned: 'exec' with pid 69
28/03/2017 14:29:122017-03-28 12:29:11,983 INFO spawned: 'pxdaemon' with pid 70
28/03/2017 14:29:122017-03-28 12:29:12,232 INFO spawned: 'px_event_listener' with pid 71
28/03/2017 14:29:122017-03-28 12:29:12,833 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:29:122017-03-28 12:29:12,833 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:29:122017-03-28 12:29:12,838 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:29:12Tue Mar 28 12:29:12 UTC 2017 size 4096 is within limits of maxsize 436207616
28/03/2017 14:29:122017-03-28 12:29:12,950 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:29:13PXPROCS: lttng not started yet...sleeping
28/03/2017 14:29:13
28/03/2017 14:29:132017-03-28 12:29:13,029 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
28/03/2017 14:29:16PXPROCS: lttng not started yet...sleeping
8/03/2017 14:29:16PXPROCS: lttng not started yet...sleeping
28/03/2017 14:29:16
28/03/2017 14:29:19PXPROCS: lttng not started yet...sleeping
28/03/2017 14:29:19
28/03/2017 14:29:19Spawning a session daemon
28/03/2017 14:29:19Session pxd created.
28/03/2017 14:29:19Traces will be written in net://localhost
28/03/2017 14:29:19Live timer set to 2000000 usec
28/03/2017 14:29:19Session pxd set to shm_path: /var/lib/osd/lttng/pxd-20170328-122919.
28/03/2017 14:29:19UST channel pxd_channel enabled for session pxd
28/03/2017 14:29:20All UST events are enabled in channel pxd_channel
28/03/2017 14:29:20Tracing started for session pxd
28/03/2017 14:29:22PXPROCS: Started px-ns with pid 123
28/03/2017 14:29:22PXPROCS: Started px-storage with pid 124
28/03/2017 14:29:22bash: connect: Connection refused
28/03/2017 14:29:22bash: /dev/tcp/localhost/9009: Connection refused
28/03/2017 14:29:22PXPROCS: px-storage not started yet...sleeping
28/03/2017 14:29:22
28/03/2017 14:29:22time="2017-03-28T12:29:22Z" level=info msg="px-ns Starting.."
28/03/2017 14:29:22NS client starting fuse module
28/03/2017 14:29:22Starting NS server
28/03/2017 14:29:22C++ grpc server listening on 0.0.0.0:9009
28/03/2017 14:29:25PXPROCS: Started px with pid 145
28/03/2017 14:29:25PXPROCS: Started watchdog with pid 146
28/03/2017 14:29:252017-03-28_12:29:25: PX-Watchdog: Starting watcher
28/03/2017 14:29:262017-03-28_12:29:26: PX-Watchdog: Waiting for px process to start
28/03/2017 14:29:27root       145    70 58 12:29 ?        00:00:01 /usr/local/bin/px -daemon
28/03/2017 14:29:272017-03-28_12:29:27: PX-Watchdog: (pid 145): Begin monitoring
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Registering [kernel] as a volume driver"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Starting PX Version: 1.1.6-693c3b5 - Build Version 693c3b5a7b1dbc9eda8f2bb7e07c44455c94aafe"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Developer license detected"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Initializing PX hooks with none"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=error msg="Could not initialize scheduler hooks for none: Unhandled scheduler "
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Loaded existing NodeID: 47765f32-d8ae-44b9-80e2-b98feea1037d"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Configuration Loaded..."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Cluster ID: 14c12ce6-13a7-11e7-93ae-92361f002671"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Node ID: 47765f32-d8ae-44b9-80e2-b98feea1037d"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Node Index: 3"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Management Iface: "
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX KVDB: [http://10.42.226.249:2379]"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Logging API: "
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX Storage Devices: [/dev/xvdc], Raid Level: data() md()"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX initialized with Docker host unix:///var/run/docker.sock"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Alerts initialized successfully for this cluster"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX starting cluster manager..."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX cluster manager running."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX limiting cluster size to 3."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Starting server on port: 9001"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="PX API server running on port 9001."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Adding cluster event listener: PX Storage Service"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Adding cluster event listener: Scheduler"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Cluster manager starting..."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Cluster state is OK... Joining the cluster."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Node 47765f32-d8ae-44b9-80e2-b98feea1037d joining cluster..."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Cluster ID: 14c12ce6-13a7-11e7-93ae-92361f002671"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Node Mgmt IP: 172.20.20.49"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=info msg="Node Data IP: 172.20.20.49"
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=error msg="This node is already initialized but could not be found in the cluster map."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=error msg="Failed to initialize node in cluster. This node is already initialized but could not be found in the cluster map."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=error msg="Init Failure: This node is already initialized but could not be found in the cluster map. Please remove /etc/pwx/.private.json to initialize a new node."
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=warning msg="Not found: /v1/osd-volumes "
28/03/2017 14:29:27time="2017-03-28T12:29:27Z" level=error msg="Error while calling home: HTTP-404: 404 page not found\n"

Almost looks like it is trying to re-initialize the node instead of re-joining the cluster.

Setup: can't get k/v to work properly

Hi, sorry, kind of new to this:

I want to test encrypted volumes but can't get the k/v store to work properly. I have etcd running:

/tmp/etcd-download-test/etcdctl --version
etcdctl version: 3.3.1
API version: 2

and I'm using the docker image from https://docs.portworx.com/developer/index.html for simplicity, although I've also tried the setup without docker. My portworx flags in the docker command look like

-k etcd://127.0.0.1:2379 -c MY_CLUSTER_ID -s /dev/sda

The docs say I should make sure that clusterid is "unique within your etcd key/value space" - I'm not sure what this means, do I need to use etcdctl mkdir or set (or put with ETCDCTL_API=3) with the cluster's name and a value?

When I set the cluster wide secret key it tells me it is properly set, but I see nothing in etcdctl. Output on the portworx side:

time="2018-02-19 09:38:48" level=info msg="Attempting Secrets Login to KVDB endpoint..." 
time="2018-02-19 09:38:53" level=info msg="Watch cb for key pwx/MY_CLUSTER_ID/ returned err: Stopped watch" 
time="2018-02-19 09:38:53" level=error msg="Watch for pwx/MY_CLUSTER_ID/ stopped" 
time="2018-02-19 09:38:53" level=info msg="Cluster Key Set successfully!" 

Using /opt/pwx/bin/pxctl volume create --secure --size 10 encrypted_volume just tells me

create: Volume (Name: encrypted_volume) create failed, error: Key not found

PX starts in maintenance mode

$ /opt/pwx/bin/pxctl status
Status: PX is in maintenance Mode. Use the service mode option to repair the system
License: PX-Developer
Node ID: 29a24ea7-a84d-4142-a412-b7fb83fbc695
IP: 192.168.10.68
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 HIGH raid0 200 GiB 0 B Offline default default
Local Storage Devices: 2 devices
Device Path Media Type Size Last-Scan
0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 100 GiB 11 Jul 17 01:30 UTC
0:2 /dev/sdc STORAGE_MEDIUM_MAGNETIC 100 GiB 11 Jul 17 01:30 UTC
total - 200 GiB
Cluster Summary
Cluster ID: devops-cluster-01
Cluster UUID: 59bbd96e-a802-4718-94cd-82fb74a22ae9
IP ID Used Capacity Status
192.168.10.68 29a24ea7-a84d-4142-a412-b7fb83fbc695 Unavailable Unavailable In Maintenance (This node)
Global Storage Pool
Total Used : 0 B
Total Capacity : 0 B

AlertID Resource ResourceID Timestamp Severity AlertType Description
2 NODE 29a24ea7-a84d-4142-a412-b7fb83fbc695 Jul 11 01:30:36 UTC 2017 ALARM Cluster manager failure Could not start cluster manager because of: Failed to mount device UUID e7987d9c-b2e1-4a35-9718-fe7981045fb6, dev path /dev/sdc at /var/.px/0: error no such device

$ sudo -i blkid
/dev/sdb: UUID="e7987d9c-b2e1-4a35-9718-fe7981045fb6" UUID_SUB="8fa72d6c-aa94-4c20-bc36-a8cbcd9f59a9" TYPE="btrfs"
/dev/sdc: UUID="e7987d9c-b2e1-4a35-9718-fe7981045fb6" UUID_SUB="fa5a359e-3da6-4b18-97aa-b78a3f439118" TYPE="btrfs"

kubernetes pvc fails with hostname empty

I am following this guide to deploy portworx. I was able to run the portworx pod just fine, and the status looks like:

root@kubeguest04:/# /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: a6247594-f908-4789-8998-9657682b027d
	IP: 10.129.37.35
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	HIGH		raid0		441 GiB	2.0 GiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path			Media Type		Size		Last-Scan
	0:1	/dev/mapper/cl-home	STORAGE_MEDIUM_MAGNETIC	441 GiB		20 Jun 17 01:04 UTC
	total				-			441 GiB
Cluster Summary
	Cluster ID: portworx-storage-0
	IP		ID					Used	Capacity	Status
	10.129.4.35	a6247594-f908-4789-8998-9657682b027d	0 B	441 GiB		Online (This node)
Global Storage Pool
	Total Used    	:  0 B
	Total Capacity	:  441 GiB

[root@kubeguest04 ~]# kubectl exec -it portworx-storage-x31h9  -- /opt/pwx/bin/pxctl cluster alerts
AlertID	ClusterID		Timestamp			Severity	AlertType		Description
0	portworx-storage-0	Jun 20 01:04:23 UTC 2017	NOTIFY		Node start success	Node a6247594-f908-4789-8998-9657682b027d with Index (0) is Up
1	portworx-storage-0	Jun 20 01:04:38 UTC 2017	NOTIFY		Node start success	Node a6247594-f908-4789-8998-9657682b027d joining the cluster with index (0)
2	portworx-storage-0	Jun 20 01:04:43 UTC 2017	NOTIFY		Node start success	PX is ready on Node: a6247594-f908-4789-8998-9657682b027d. CLI accessible at /opt/pwx/bin/pxctl.

[root@kubeguest04 ~]# kubectl create -f pvc.yml
persistentvolumeclaim "minio-persistent-storage" created

Now, when i tried to create a pvc, it fails with:

[root@kubeguest04 ~]# kubectl describe pvc minio-persistent-storage
Name:		minio-persistent-storage
Namespace:	default
StorageClass:	portworx
Status:		Pending
Volume:
Labels:		<none>
Annotations:	volume.beta.kubernetes.io/storage-class=portworx
		volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/portworx-volume
Capacity:
Access Modes:
Events:
  FirstSeen	LastSeen	Count	From				SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----				-------------	--------	------		-------
  22s		6s		3	persistentvolume-controller			Warning		ProvisioningFailed	Failed to provision volume with StorageClass "portworx": Post http://:9001/v1/osd-volumes: dial tcp :9001: getsockopt: connection refused

Why is it connecting to http://:9001 and not including the hostname in there.

Though, when i tried on hostname, i am able to connect just fine:

[root@kubeguest04 ~]# curl http://10.129.4.35:9001/v1/osd-volumes
[]

ERROR: for portworx Cannot start service portworx: linux mounts: Path /opt/pwx/bin is mounted on / but it is not a shared mount

Greetings,

Trying to get px-get working. Followed the quick-start guide, however encountered problems.

Environment:

NAME="Red Hat Enterprise Linux Server"
VERSION="7.2 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.2"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.2 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.2"

Docker info:

Containers: 3
 Running: 2
 Paused: 0
 Stopped: 1
Images: 42
Server Version: 1.12.1
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rexray
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.4.4.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.2 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.58 GiB
Name: local1
ID: 2CLA:UCHL:R5J4:IFLP:UOYW:HUVM:DT4M:FUSP:UX6I:SQTB:67PH:5YIN
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: twistlockreader
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8

File: config.json

{
  "clusterid": "d6a1096b-3ba6-4e1e-b75b-b7f7a8fa6df7"
  "kvdb": [ 
              "etcd:http://127.0.0.1:2379"
         ],
  "storage": {
    "devices": [
      "/dev/sdh"
    ]
  }
}

Error:

[root@node261 quick-start]# docker-compose up -d 
Pulling portworx (portworx/px-dev:latest)...
latest: Pulling from portworx/px-dev
a2392627bec4: Pull complete
6f75e6962b30: Pull complete
f5e1684bcd4b: Pull complete
1af17c678254: Pull complete
b21c7216d97e: Pull complete
ba99d4dc1b96: Pull complete
c93ccce1ede9: Pull complete
0d377a9f3876: Pull complete
4d1e825eea0e: Pull complete
1fb846e844bb: Pull complete
a1df505be760: Pull complete
89407dacbba6: Pull complete
c823797e1d88: Pull complete
Digest: sha256:6c1c58b06e1efe7f198319b998d758874e35db0bf4287ceb5b51a7dfe92792c3
Status: Downloaded newer image for portworx/px-dev:latest
Creating px-dev

ERROR: for portworx  Cannot start service portworx: linux mounts: Path /opt/pwx/bin is mounted on / but it is not a shared mount.
ERROR: Encountered errors while bringing up the project.
[root@node261 quick-start]#

Any ideas why is it bombing out ?

Alex

Recomended AWS instance type

What is the recomended AWS instance type to use with portworx?

etcd needs for px-dev on kubernetes

Does px-dev on kubernetes need etcd?
It is not mentioned in https://github.com/portworx/px-dev/blob/master/install_with_k8s.md

Volume will not unmount

$ pxctl status
Status: PX is operational
License: Trial (expires in 22 days)
Node ID: 5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71
IP: 192.168.10.68
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 HIGH raid0 100 GiB 2.9 GiB Online default default
Local Storage Devices: 1 device
Device Path Media Type Size Last-Scan
0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 100 GiB 20 Jul 17 15:47 UTC
total - 100 GiB
Cluster Summary
Cluster ID: devops-cluster-01
Cluster UUID: 5238855239480218275
IP ID Used Capacity Status
192.168.10.68 5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71 2.9 GiB 100 GiB Online (This node)
192.168.10.138 1a47ccf0-95d8-4964-a4e1-3f0adbe6e707 2.7 GiB 100 GiB Online
Global Storage Pool
Total Used : 5.6 GiB
Total Capacity : 200 GiB

~$ pxctl v l
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY SCALE STATUS
105555218929839946 jenkins-home 10 GiB 1 no no LOW 1 up - attached on 192.168.10.68
1081000382262830454 jenkins-slave-ssh 1 GiB 2 yes no LOW 1 up - attached on 192.168.10.138
1013395067276956434 m2-repository 4 GiB 2 yes no LOW 1 up - attached on 192.168.10.68

$ pxctl v d 1013395067276956434
delete: Volume ID 1013395067276956434 is attached on IP 192.168.10.68 (5de69d6c-1cff-4ae1-8b37-e7f4e89c7f71). Please stop the Docker container using the volume first.

==== second host =====
$ pxctl host attach 1013395067276956434
Volume successfully attached at: /dev/pxd/pxd1013395067276956434
abienkow@adrian-vm:~$ pxctl v l
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY SCALE STATUS
105555218929839946 jenkins-home 10 GiB 1 no no LOW 1up - attached on 192.168.10.68
1081000382262830454 jenkins-slave-ssh 1 GiB 2 yes no LOW 1up - attached on 192.168.10.138
1013395067276956434 m2-repository 4 GiB 2 yes no LOW 1up - attached on 192.168.10.68

==== back to first host ===
$ pxctl v d 1013395067276956434
delete: Volume 1013395067276956434 is mounted at 1 location(s): /var/lib/osd/pxns/1013395067276956434
$ pxctl host unmount 1013395067276956434 --path /var/lib/osd/pxns/1013395067276956434
Volume 1013395067276956434 successfully unmounted at /var/lib/osd/pxns/1013395067276956434
$ pxctl v d 1013395067276956434 delete: Volume 1013395067276956434 is mounted at 1 location(s): /var/lib/osd/pxns/1013395067276956434

Replacing a filed disk

In my scenario, my virtualized local disk was removed from one of the nodes, and I replaced it with another volume.

px driver starts in maintenance mode, because it can not find the original /dev/sdd device with a specific uuid. I've added a new device to replace it, and expected the process to start initializing the new device and replicating data from the second node.

I've also tried pxctl sv drive add /dev/sdc, but it keeps failing because of maintenance mode.

Is there another step that is needed, --recover to exit maintenance mode does not help to recover the situation.

px-dev 1.2 broken on Ubuntu 16.04

When running px-dev 1.2 on Ubuntu 16.04, I'm getting the following error:

Apr 22 21:53:00 Sat Apr 22 19:53:00 UTC 2017 : Running on Linux kube3 4.10.8-docker-1 #1 SMP Tue Apr 4 16:18:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Apr 22 21:53:00 checking /hostusr/src/kernels/4.10.8-docker-1
Apr 22 21:53:00 checking /hostusr/src/linux-headers-4.10.8-docker-1
Apr 22 21:53:00 checking /usr/src/kernels/4.10.8-docker-1
Apr 22 21:53:00 checking /usr/src/linux-headers-4.10.8-docker-1
Apr 22 21:53:00 checking /lib/modules/4.10.8-docker-1/build
Apr 22 21:53:00 checking /tmp/kernel_headers/usr/src/linux-headers-4.10.8-docker-1
Apr 22 21:53:00 checking /usr/src/linux
Apr 22 21:53:00 found /usr/src/linux
Apr 22 21:53:00 checking /usr/src/linux
Apr 22 21:53:00 found /usr/src/linux
Apr 22 21:53:00 Flushing buffer-cache..
Apr 22 21:53:00 checking /usr/src/linux
Apr 22 21:53:00 found /usr/src/linux
Apr 22 21:53:00 make -C /usr/src/linux  M=/home/px-fuse clean
Apr 22 21:53:00 make[1]: Entering directory '/usr/src/linux-4.10.8'
Apr 22 21:53:01 make[1]: Leaving directory '/usr/src/linux-4.10.8'
Apr 22 21:53:01 make -C /usr/src/linux  M=/home/px-fuse modules
Apr 22 21:53:02 make[1]: Entering directory '/usr/src/linux-4.10.8'
Apr 22 21:53:03   CC [M]  /home/px-fuse/pxd.o
Apr 22 21:53:04 /home/px-fuse/pxd.c: In function 'pxd_make_request':
Apr 22 21:53:04 /home/px-fuse/pxd.c:386:13: error: implicit declaration of function 'bio_flags' [-Werror=implicit-function-declaration]
Apr 22 21:53:04   op_flags = bio_flags(bio);
Apr 22 21:53:04              ^
Apr 22 21:53:05 cc1: all warnings being treated as errors
Apr 22 21:53:05 make[2]: *** [/home/px-fuse/pxd.o] Error 1
Apr 22 21:53:05 scripts/Makefile.build:294: recipe for target '/home/px-fuse/pxd.o' failed
Apr 22 21:53:05 Makefile:1490: recipe for target '_module_/home/px-fuse' failed
Apr 22 21:53:05 make[1]: *** [_module_/home/px-fuse] Error 2
Apr 22 21:53:05 make[1]: Leaving directory '/usr/src/linux-4.10.8'
Apr 22 21:53:05 make: *** [all] Error 2
Apr 22 21:53:05 Makefile:45: recipe for target 'all' failed
Apr 22 21:53:05 Failed to load PX module dependencies
Apr 22 21:53:34 Sat Apr 22 19:53:34 UTC 2017 : Running on Linux kube3 4.10.8-docker-1 #1 SMP Tue Apr 4 16:18:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Apr 22 21:53:34 checking /hostusr/src/kernels/4.10.8-docker-1
Apr 22 21:53:34 checking /hostusr/src/linux-headers-4.10.8-docker-1
Apr 22 21:53:34 checking /usr/src/kernels/4.10.8-docker-1
Apr 22 21:53:34 checking /usr/src/linux-headers-4.10.8-docker-1
Apr 22 21:53:34 checking /lib/modules/4.10.8-docker-1/build
Apr 22 21:53:34 checking /tmp/kernel_headers/usr/src/linux-headers-4.10.8-docker-1
Apr 22 21:53:34 checking /usr/src/linux
Apr 22 21:53:34 found /usr/src/linux
Apr 22 21:53:34 checking /usr/src/linux
Apr 22 21:53:34 found /usr/src/linux
Apr 22 21:53:34 Flushing buffer-cache..
Apr 22 21:53:34 checking /usr/src/linux
Apr 22 21:53:34 found /usr/src/linux
Apr 22 21:53:34 make -C /usr/src/linux  M=/home/px-fuse clean
Apr 22 21:53:34 make[1]: Entering directory '/usr/src/linux-4.10.8'
Apr 22 21:53:35 make[1]: Leaving directory '/usr/src/linux-4.10.8'
Apr 22 21:53:35 make -C /usr/src/linux  M=/home/px-fuse modules
Apr 22 21:53:35 make[1]: Entering directory '/usr/src/linux-4.10.8'
Apr 22 21:53:37   CC [M]  /home/px-fuse/pxd.o
Apr 22 21:53:38 /home/px-fuse/pxd.c: In function 'pxd_make_request':
Apr 22 21:53:38 /home/px-fuse/pxd.c:386:13: error: implicit declaration of function 'bio_flags' [-Werror=implicit-function-declaration]
Apr 22 21:53:38   op_flags = bio_flags(bio);
Apr 22 21:53:38              ^
Apr 22 21:53:39 cc1: all warnings being treated as errors
Apr 22 21:53:39 make[2]: *** [/home/px-fuse/pxd.o] Error 1
Apr 22 21:53:39 scripts/Makefile.build:294: recipe for target '/home/px-fuse/pxd.o' failed
Apr 22 21:53:39 Makefile:1490: recipe for target '_module_/home/px-fuse' failed
Apr 22 21:53:39 make[1]: Leaving directory '/usr/src/linux-4.10.8'
Apr 22 21:53:39 Makefile:45: recipe for target 'all' failed
Apr 22 21:53:39 make[1]: *** [_module_/home/px-fuse] Error 2
Apr 22 21:53:39 make: *** [all] Error 2
Apr 22 21:53:39 Failed to load PX module dependencies 

1.1.6 runs fine

linux-headers "no such file or directory"

Environment

• Debian stretch (4.9.0-4-amd64)
• Docker CE (17.09.0~ce-0~debian)

Docker Info

Containers: 13
Running: 5
Paused: 0
Stopped: 8
Images: 8
Server Version: 17.09.0-ce
Storage Driver: devicemapper
Pool Name: vg0-docker_thin
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: ext4
Data file:
Metadata file:
Data Space Used: 5.097GB
Data Space Total: 161.1GB
Data Space Available: 156GB
Metadata Space Used: 2.429MB
Metadata Space Total: 7.516GB
Metadata Space Available: 7.514GB
Thin Pool Minimum Free Space: 16.11GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.137 (2016-11-30)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local lvm
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
seccomp
 Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.56GiB
Name: vm-box
ID: K4YD:7C65:AY5M:MNKA:K6OV:CKJ4:WEBH:LKKV:WILX:U4OW:6PJN:OOSC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

I installed the portworx plugin, following the steps at https://docs.portworx.com/scheduler/docker/docker-plugin.html and I tried to use the docker-compose.yml from https://github.com/portworx/px-dev/blob/master/quick-start/docker-compose.yml

Both versions are throwing this execptions:

Creating px fs...
make -C /usr/src/linux-headers-4.9.0-4-common O=/usr/src/linux-headers-4.9.0-4-amd64 M=/home/px-fuse clean
make[1]: Entering directory '/usr/src/linux-headers-4.9.0-4-common'
make[2]: Entering directory '/usr/src/linux-headers-4.9.0-4-amd64'
/usr/src/linux-headers-4.9.0-4-common/Makefile:308: scripts/Kbuild.include: No such file or directory
/bin/bash: /usr/src/linux-headers-4.9.0-4-common/scripts/gcc-goto.sh: No such file or directory
/usr/src/linux-headers-4.9.0-4-common/Makefile:667: scripts/Makefile.gcc-plugins: No such file or directory
/usr/src/linux-headers-4.9.0-4-common/Makefile:812: scripts/Makefile.kasan: No such file or directory
/usr/src/linux-headers-4.9.0-4-common/Makefile:813: scripts/Makefile.extrawarn: No such file or directory
/usr/src/linux-headers-4.9.0-4-common/Makefile:814: scripts/Makefile.ubsan: No such file or directory

I already purged and reinstalled all headers and kbuild packages. Nothing changed.

Entering multiple etcd hosts in Rancher config

We are using Rancher with Docker, and using a slightly customized version of the community catalog px-dev entry.

For the etcd / consul server, it looks like you can only enter one etcd host, while the config.json file that is created during the Rancher spinup uses a json array for this and therefore supports multiple instances.

I have tried entering a comma separated list - but this ends up with an incorrect entry in the kvdb section of the config.json.

Increase device size

How could I increase an AWS EC2 EBS drive in portworx that has increased in AWS?

I've build a cluster with 3 docker swarm nodes and installed portworx as a docker plugin. I've attached, for testing purposes, 2x 1GB EBS backed disks in each node. This is the cluster status:

root@swarm-manager-001:/tmp/stacks# /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: 621ff80d-c47f-42e0-9ab2-fc67dba0b688
	IP: 10.0.1.101 
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	MEDIUM		raid0		2.0 GiB	225 MiB	Online	a	sa-east-1
	Local Storage Devices: 2 devices
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/xvdg	STORAGE_MEDIUM_MAGNETIC	1.0 GiB		11 Jul 17 12:44 UTC
	0:2	/dev/xvdh	STORAGE_MEDIUM_MAGNETIC	1.0 GiB		11 Jul 17 12:44 UTC
	total			-			2.0 GiB
Cluster Summary
	Cluster ID: socialbase
	IP		ID					Used	Capacity	Status
	10.0.1.101	621ff80d-c47f-42e0-9ab2-fc67dba0b688	225 MiB	2.0 GiB		Online (This node)
	10.0.2.102	31a1ff1f-4e12-4a73-9fae-9009ce855144	225 MiB	2.0 GiB		Online
	10.0.3.103	b6505999-5415-422e-9b89-c9dac8e6bf51	225 MiB	2.0 GiB		Online
Global Storage Pool
	Total Used    	:  676 MiB
	Total Capacity	:  6.0 GiB

I've extended the size of 4 EBS disks to 2GB. Two of the 4 resized EBS disks are attached to node 10.0.1.101 and the other 2 are attached to 10.0.2.102 and 10.0.3.103:

Now I need to reflect the resize action that I did in AWS EC2 EBS console in the portworx cluster. How could I do it?

References:
http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ebs-expand-volume.html#console-modify
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html#recognize-expanded-volume-linux

Can not install PX on CoreOs

I've use the following doc:

https://docs.portworx.com/scheduler/docker/docker-plugin.html#upgrading-portworx-container-to-portworx-v2-docker-plugin

sudo docker plugin install portworx/px:latest --alias pxd

Problem is the: - mount: [/usr/src]

Within CoreOs that volume is read-only, so not usable.

I've found the follwing line inside the px-mon repo:
https://github.com/portworx/px-installer/blob/c3c0acb5c23584207d493f91b27d5cb57ffa99da/px-mon/px-mon.go#L257

However, seems not to work, any help? :)

"pxctl service pool" not exists

The sub command pool of command service not exists:

root@swarm-manager-001:~# /opt/pwx/bin/pxctl service pool show
No help topic for 'pool'

Cluster info

root@swarm-manager-001:~# /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: 621ff80d-c47f-42e0-9ab2-fc67dba0b688
	IP: 10.0.1.101 
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	MEDIUM		raid0		2.0 GiB	225 MiB	Online	a	sa-east-1
	Local Storage Devices: 2 devices
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/xvdg	STORAGE_MEDIUM_MAGNETIC	1.0 GiB		11 Jul 17 13:31 UTC
	0:2	/dev/xvdh	STORAGE_MEDIUM_MAGNETIC	1.0 GiB		11 Jul 17 13:31 UTC
	total			-			2.0 GiB
Cluster Summary
	Cluster ID: socialbase
	IP		ID					Used	Capacity	Status
	10.0.1.101	621ff80d-c47f-42e0-9ab2-fc67dba0b688	225 MiB	2.0 GiB		Online (This node)
	10.0.2.102	31a1ff1f-4e12-4a73-9fae-9009ce855144	225 MiB	2.0 GiB		Online
	10.0.3.103	b6505999-5415-422e-9b89-c9dac8e6bf51	0 B	0 B		Online
Global Storage Pool
	Total Used    	:  451 MiB
	Total Capacity	:  4.0 GiB

oot@swarm-manager-001:~# lsblk 
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   12G  0 disk 
└─xvda1 202:1    0   12G  0 part /
xvdf    202:80   0    1G  0 disk 
└─xvdf1 202:81   0 1023M  0 part 
xvdg    202:96   0    2G  0 disk 
xvdh    202:112  0    2G  0 disk 
xvdi    202:128  0   30G  0 disk 

root@swarm-manager-001:~# docker plugin ls
ID                  NAME                DESCRIPTION                         ENABLED
debb3f661bf6        pxd:latest          Portworx Data Services for Docker   true


root@swarm-manager-001:~# docker plugin inspect pxd:latest
[
    {
        "Config": {
            "Args": {
                "Description": "Portworx Volume Driver Options. Refer http://docs.portworx.com for a full list of supported options",
                "Name": "opts",
                "Settable": [
                    "value"
                ],
                "Value": null
            },
            "Description": "Portworx Data Services for Docker",
            "Documentation": "http://docs.portworx.com/",
            "Entrypoint": [
                "/docker-entry-point.sh",
                "-daemon"
            ],
            "Env": [
                {
                    "Description": "",
                    "Name": "GOTRACEBACK",
                    "Settable": null,
                    "Value": "crash"
                },
                {
                    "Description": "",
                    "Name": "GOMAXPROCS",
                    "Settable": null,
                    "Value": "64"
                },
                {
                    "Description": "",
                    "Name": "PXMOD_SOURCE",
                    "Settable": null,
                    "Value": "/home/px-fuse"
                },
                {
                    "Description": "",
                    "Name": "PXMOD_VERSION",
                    "Settable": null,
                    "Value": "2"
                },
                {
                    "Description": "",
                    "Name": "PX_PLUGIN",
                    "Settable": null,
                    "Value": "true"
                }
            ],
            "Interface": {
                "Socket": "pxd.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": true,
                "Capabilities": [
                    "CAP_IPC_LOCK",
                    "CAP_SYS_ADMIN",
                    "CAP_SYS_MODULE"
                ],
                "Devices": null
            },
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/etc/pwx",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/etc/pwx",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/lib/osd",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/lib/osd",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/export_bin",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/opt/pwx/bin",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/run/docker.sock",
                    "Name": "",
                    "Options": [
                        "shared",
                        "bind"
                    ],
                    "Settable": null,
                    "Source": "/var/run/docker.sock",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/lib/modules",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/lib/modules",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/usr/src",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/usr/src",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/cores",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/cores",
                    "Type": "bind"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/var/lib/osd",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:d9519377a52707e2347220a92802441842ad70c6936798d20f60d7c4997ce3a1"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "debb3f661bf6dbaf494ff474e600a721535ad4750ebf1546b1b544c5cffed5cb",
        "Name": "pxd:latest",
        "PluginReference": "docker.io/portworx/px:latest",
        "Settings": {
            "Args": [
                "-k",
                "consul:http://myconsul:8500",
                "-c",
                "socialbase",
                "-s",
                "/dev/xvdg",
                "-s",
                "/dev/xvdh"
            ],
            "Devices": [],
            "Env": [
                "GOTRACEBACK=crash",
                "GOMAXPROCS=64",
                "PXMOD_SOURCE=/home/px-fuse",
                "PXMOD_VERSION=2",
                "PX_PLUGIN=true"
            ],
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/etc/pwx",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/etc/pwx",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/lib/osd",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/lib/osd",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/export_bin",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/opt/pwx/bin",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/run/docker.sock",
                    "Name": "",
                    "Options": [
                        "shared",
                        "bind"
                    ],
                    "Settable": null,
                    "Source": "/var/run/docker.sock",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/lib/modules",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/lib/modules",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/usr/src",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/usr/src",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/cores",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/cores",
                    "Type": "bind"
                }
            ]
        }
    }
]

max-count parameter

The docs aren't clear about max-count parameter: https://docs.portworx.com/cloud/aws/asg.html#scaling-the-cluster-up. Where I need to set it? I mean, what is its correct syntax? Is it a command line parameter or a configuration that needs to be set through environment variable? Is it works with px-dev?

404 Not Found during download of kernel headers for 4.4.59-boot2docker

Probably similar to #16, I'm having problems running px-dev on my docker-machine.

During launch of the container, it looks for http://mirrors.portworx.com/build-results/pxfuse/for-installer/x86_64/4.4.59-boot2docker/packages; but it fails, when I checked the URL at my browser, I get 404 not found.

docker version:

Client:
Version: 17.04.0-ce
API version: 1.28
Go version: go1.7.5
Git commit: 4845c56
Built: Wed Apr 5 18:45:47 2017
OS/Arch: linux/amd64

Server:
Version: 17.04.0-ce
API version: 1.28 (minimum version 1.12)
Go version: go1.7.5
Git commit: 4845c56
Built: Wed Apr 5 18:45:47 2017
OS/Arch: linux/amd64
Experimental: true

docker-machine version:

docker-machine.exe version 0.10.0, build 76ed2a6

log for the container:

Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-headers-4.4.59-boot2docker
E: Couldn't find any package by glob 'linux-headers-4.4.59-boot2docker'
E: Couldn't find any package by regex 'linux-headers-4.4.59-boot2docker'
checking /hostusr/src/kernels/4.4.59-boot2docker
checking /hostusr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/kernels/4.4.59-boot2docker
checking /usr/src/linux-headers-4.4.59-boot2docker
checking /lib/modules/4.4.59-boot2docker/build
checking /tmp/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/linux
checking /tmp/px_workspace/kernel_headers/usr/src/kernels/4.4.59-boot2docker
checking /tmp/px_workspace/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /hostusr/src/kernels/4.4.59-boot2docker
checking /hostusr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/kernels/4.4.59-boot2docker
checking /usr/src/linux-headers-4.4.59-boot2docker
checking /lib/modules/4.4.59-boot2docker/build
checking /tmp/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/linux
checking /tmp/px_workspace/kernel_headers/usr/src/kernels/4.4.59-boot2docker
checking /tmp/px_workspace/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /hostusr/src/kernels/4.4.59-boot2docker
checking /hostusr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/kernels/4.4.59-boot2docker
checking /usr/src/linux-headers-4.4.59-boot2docker
checking /lib/modules/4.4.59-boot2docker/build
checking /tmp/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/linux
checking /tmp/px_workspace/kernel_headers/usr/src/kernels/4.4.59-boot2docker
checking /tmp/px_workspace/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /hostusr/src/kernels/4.4.59-boot2docker
checking /hostusr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/kernels/4.4.59-boot2docker
checking /usr/src/linux-headers-4.4.59-boot2docker
checking /lib/modules/4.4.59-boot2docker/build
checking /tmp/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/linux
checking /tmp/px_workspace/kernel_headers/usr/src/kernels/4.4.59-boot2docker
checking /tmp/px_workspace/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /hostusr/src/kernels/4.4.59-boot2docker
checking /hostusr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/kernels/4.4.59-boot2docker
checking /usr/src/linux-headers-4.4.59-boot2docker
checking /lib/modules/4.4.59-boot2docker/build
checking /tmp/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
checking /usr/src/linux
checking /tmp/px_workspace/kernel_headers/usr/src/kernels/4.4.59-boot2docker
checking /tmp/px_workspace/kernel_headers/usr/src/linux-headers-4.4.59-boot2docker
Downloading from: http://mirrors.portworx.com/build-results/pxfuse/for-installer/x86_64/4.4.59-boot2docker/packages...
Download linux-kbuild failed...
Failed to install/locate kernel headers for 4.4.59-boot2docker
Failed to load PX module dependencies

Any suggestions?

RAID question

Quick question as the documentation doesn't seem to shed any light on this. I am a little worried that it says raid0. The data is replicated to two nodes when the shared volume was created so surely this should say RAID1?

I just want to make sure that my data is safe. Is there any way to change this to RAID1?

Status: PX is operational
License: PX-Developer
Node ID: 48f1b703-6d6e-4cea-b7f0-0a4eb7294b95
	IP: 172.30.1.220
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	HIGH		raid0		20 GiB	276 MiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/xvdb	STORAGE_MEDIUM_SSD	20 GiB		06 Dec 17 18:02 UTC
	total			-			20 GiB
Cluster Summary
	Cluster ID: ef4d3327-ad43-42f8-a419-f5b76fd95423
	Cluster UUID: a68955bc-d6e9-480b-9f9f-92a7f2faf44d
	Nodes: 3 node(s) with storage (3 online)
	IP		ID					StorageNode	Used	Capacity	Status
	172.30.3.220	a9fdadb3-558c-425f-995a-08e14b6d855c	Yes		1.2 GiB	100 GiB		Online
	172.30.2.220	a73ec252-3ec3-452e-b7f1-649943381e81	Yes		1.3 GiB	100 GiB		Online
	172.30.1.220	48f1b703-6d6e-4cea-b7f0-0a4eb7294b95	Yes		0 B	20 GiB		Online	 (This node)

recover node after reboot

Hi there,

after rebooting my px-dev host can't start the storage and is stuck in "initializing". Is there any way to force quorum and enable / start this host?

I'm running px version: pxctl version 1.1.6-cb1bbeb

In the px-dev container I get some warning about locks and after w while the container restarts

time="2017-04-26T20:46:11Z" level=warning msg="Lock pwx/5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a7/storage/locks/1056637145994285341.lock locked for 285 seconds, tag: {NodeID:94e7b4e6-6bf8-41a5-bfa0-45bfe6afdd22,FuncID:postVolumeUsage}"

time="2017-04-26T20:46:25Z" level=warning msg="Failed to acquire kvdb lock: Key already exists" Error="Key already exists" Function=volumeStateHandler Notification=&{281474976710698 751556 {1056637145994285341 0 0 2 [0xc8209b6c80] false false map[]}}
time="2017-04-26T20:46:25Z" level=warning msg="VolumeState: {281474976710698 751556 {1056637145994285341 0 0 2 [0xc8209b6c80] false false map[]}}" Error="Failed to acquire kvdb lock: Key already exists" Function=processNotification Notification=&{64 0001-01-01 00:00:00 +0000 UTC 0 {281474976710698 751556 {1056637145994285341 0 0 2 [0xc8209b6c80] false false map[]}}}
time="2017-04-26T20:46:25Z" level=warning Driver=kernel Function=VolumeNotifyFail OpId=281474976710698 Status=-1
time="2017-04-26 20:46:25Z" level=INFO msg="void ReplicationSet::block_state_notify_failed(uint64_t) token: 281474976710698"
time="2017-04-26 20:46:25Z" level=INFO msg="update_cdb: dev: 1056637145994285341 rset: 0 node[ 0 ] curr[ 0 ] next[ 0 ] new_rset [ empty ] remove [ empty ] pool_ids [ 0 ]  new_pool_ids [ empty ]"
time="2017-04-26T20:46:25Z" level=info msg="volumeStateHandler update" AbortOnError=false BackgroundProcessing=false Driver=pxd Error=<nil> Format=FS_TYPE_EXT4 Function=d.volumePut ID=657389255672230477 State=VOLUME_STATE_ATTACHED Version=751393
time="2017-04-26T20:46:25Z" level=info msg="Action: 2 data <nil>" AttachedOn=94e7b4e6-6bf8-41a5-bfa0-45bfe6afdd22 Driver=kernel Error=<nil> Function=VolumeStateChange ID=657389255672230477 State=VOLUME_STATE_ATTACHED Version=751675
time="2017-04-26T20:46:25Z" level=error msg="Failed to acquire kvdb lock Key already exists" Driver=pxd Error=<nil> Function=refreshAttachInfo
time="2017-04-26T20:46:25Z" level=error msg="Unable to start node.  Error while loading volume pxd because of: Key already exists"
time="2017-04-26T20:46:25Z" level=warning msg="Failed to initialize Join PX Storage Service: Key already exists"
time="2017-04-26T20:46:25Z" level=error msg="Failed to join cluster. Key already exists"
time="2017-04-26T20:46:25Z" level=error msg="Could not start cluster manager because of: Key already exists"

pwxctl status says:

$ sudo /opt/pwx/bin/pxctl status
Status: PX is initializing...
Node ID: 94e7b4e6-6bf8-41a5-bfa0-45bfe6afdd22
	IP: 10.10.23.92
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	SIZE	USED	STATUS	ZONE	REGION
	0	HIGH		750 GiB	9.0 GiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/loop0	STORAGE_MEDIUM_MAGNETIC	750 GiB		26 Apr 17 20:36 UTC
	total			-			750 GiB
Cluster Summary
	Cluster ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a7
	IP		ID					Used	Capacity	Status
	10.10.23.92	94e7b4e6-6bf8-41a5-bfa0-45bfe6afdd22	0 B	0 B		Initializing (This node)
Global Storage Pool
	Total Used    	:  0 B
	Total Capacity	:  0 B

cluster list:

$ sudo /opt/pwx/bin/pxctl c l
Cluster ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a7
Status: Not in Quorum

Nodes in the cluster:
ID					DATA IP		CPU	MEM TOTAL	MEM FREE	CONTAINERS	VERSION		STATUS
94e7b4e6-6bf8-41a5-bfa0-45bfe6afdd22	10.10.23.92	0.425	68 GB		65 GB		N/A		1.1.6-cb1bbeb	Initializing

Is this because it is not a 3 node cluster and therefore no quorum can be reached or is this just a side effect since pwx can not initialize the storage because of the .lock files?

Stalled at "PX is initializing"

When trying to start Portworx on a Digital Ocean droplet with an attached volume, Portworx never completes the initialisation.

Having looked at pxctl status the resulting output has a status of Status: PX is initializing....
The full output is:

Node ID: 57351339-1e43-49d6-9aa1-8799b8277d06
	IP: 207.154.208.190 
 	Local Storage Pool: 0 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	No storage pool
	Local Storage Devices: 0 device
	Device	Path	Media Type	Size		Last-Scan
	No storage device
	total		-	0 B
Cluster Summary
	Cluster ID: 6b358427202f19c3174ba14fe65b44cc43a3f5fc
	IP	ID	Used	Capacity	Status
Global Storage Pool
	Total Used    	:  0 B
	Total Capacity	:  0 B

To debug further I inspected the logs of the px-dev container:

Fri May  5 20:24:34 UTC 2017 : Running on Linux docker-512mb-fra1-01 4.4.0-75-generic #96-Ubuntu SMP Thu Apr 20 09:56:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 255967232, less than required 293601280
PXD version:  a2970b4bcdc32988b7e6520bbd7f7450b0d51241
Failed to enable rt scheduler
/docker-entry-point.sh: line 749: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Checking sysfs mount...
2017-05-05 20:24:34,440 CRIT Supervisor running as root (no user in config file)
2017-05-05 20:24:34,443 INFO supervisord started with pid 1
2017-05-05 20:24:35,448 INFO spawned: 'relayd' with pid 66
2017-05-05 20:24:35,460 INFO spawned: 'lttng' with pid 67
2017-05-05 20:24:35,489 INFO spawned: 'exec' with pid 68
2017-05-05 20:24:35,505 INFO spawned: 'pxdaemon' with pid 69
2017-05-05 20:24:35,512 INFO spawned: 'px_event_listener' with pid 77
Fri May  5 20:24:35 UTC 2017 size 1456584 is within limits of maxsize 436207616
PXPROCS: lttng not started yet...sleeping 

2017-05-05 20:24:36,596 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-05-05 20:24:36,596 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-05-05 20:24:36,597 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-05-05 20:24:36,597 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-05-05 20:24:36,597 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: lttng not started yet...sleeping 

Spawning a session daemon
Session pxd created.
Traces will be written in net://localhost
Live timer set to 2000000 usec
Session pxd set to shm_path: /var/lib/osd/lttng/pxd-20170505-202440.
UST channel pxd_channel enabled for session pxd
All UST events are enabled in channel pxd_channel
Tracing started for session pxd
PXPROCS: Started px-ns with pid 120
PXPROCS: Started px-storage with pid 121
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

time="2017-05-05T20:24:41Z" level=info msg="px-ns Starting.." 
NS client starting fuse module
Starting NS server
C++ grpc server listening on 0.0.0.0:9009
PXPROCS: Started px with pid 143
PXPROCS: Started watchdog with pid 144
2017-05-05_20:24:44: PX-Watchdog: Starting watcher
2017-05-05_20:24:44: PX-Watchdog: Waiting for px process to start
root       143    69  0 20:24 ?        00:00:00 /usr/local/bin/px -daemon
2017-05-05_20:24:44: PX-Watchdog: (pid 143): Begin monitoring
time="2017-05-05T20:24:44Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-05-05T20:24:44Z" level=info msg="Starting PX Version: 1.2.0-736cb89 - Build Version 736cb89b1184e0f95d25f13da6aaadb675e8f7a9" 
time="2017-05-05T20:24:44Z" level=info msg="Developer license detected" 
time="2017-05-05T20:24:44Z" level=info msg="Node is not yet initialized" 
time="2017-05-05T20:24:44Z" level=info msg="Generating a new Node ID and index." 
time="2017-05-05T20:24:44Z" level=info msg="New NodeID is 9abd1814-1e87-43f5-b3a8-6bf54ad1f283" 
time="2017-05-05T20:24:44Z" level=info msg="Joining cluster 6b358427202f19c3174ba14fe65b44cc43a3f5fc at this discovery service: [http://178.62.25.92:2379]." 
time="2017-05-05T20:24:44Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-05-05T20:24:44Z" level=info msg="Node 9abd1814-1e87-43f5-b3a8-6bf54ad1f283 with Index (0) is Up" 
time="2017-05-05T20:24:45Z" level=info msg="PX Configuration Loaded..." 
time="2017-05-05T20:24:45Z" level=info msg="PX Cluster ID: 6b358427202f19c3174ba14fe65b44cc43a3f5fc" 
time="2017-05-05T20:24:45Z" level=info msg="PX Node ID: 9abd1814-1e87-43f5-b3a8-6bf54ad1f283" 
time="2017-05-05T20:24:45Z" level=info msg="PX Node Index: 0" 
time="2017-05-05T20:24:45Z" level=info msg="PX Management Iface: " 
time="2017-05-05T20:24:45Z" level=info msg="PX Discovery Server(s): [http://178.62.25.92:2379]" 
time="2017-05-05T20:24:45Z" level=info msg="PX Logging API: " 
time="2017-05-05T20:24:45Z" level=info msg="PX Storage Devices: [/dev/sda], Raid Level: data() md()" 
time="2017-05-05T20:24:45Z" level=info msg="PX initialized with Docker host unix:///var/run/docker.sock" 
time="2017-05-05T20:24:45Z" level=info msg="PX starting cluster manager..." 
time="2017-05-05T20:24:45Z" level=info msg="PX cluster manager running." 
time="2017-05-05T20:24:45Z" level=info msg="PX limiting cluster size to 3." 
time="2017-05-05T20:24:45Z" level=info msg="Starting server on port: 9001" 
time="2017-05-05T20:24:45Z" level=info msg="PX API server running on port 9001." 
time="2017-05-05T20:24:45Z" level=info msg="Adding cluster event listener: PX Storage Service" 
time="2017-05-05T20:24:45Z" level=info msg="Adding cluster event listener: Scheduler" 
time="2017-05-05T20:24:45Z" level=info msg="Cluster manager starting..." 
time="2017-05-05T20:24:45Z" level=info msg="Cluster is uninitialized..." 
time="2017-05-05T20:24:45Z" level=info msg="Initializing a new cluster." 
time="2017-05-05T20:24:45Z" level=info msg="Node 9abd1814-1e87-43f5-b3a8-6bf54ad1f283 joining cluster..." 
time="2017-05-05T20:24:45Z" level=info msg="Cluster ID: 6b358427202f19c3174ba14fe65b44cc43a3f5fc" 
time="2017-05-05T20:24:45Z" level=info msg="Node Mgmt IP: 207.154.223.205" 
time="2017-05-05T20:24:45Z" level=info msg="Node Data IP: 207.154.223.205" 
time="2017-05-05T20:24:45Z" level=info msg="Service Provider information..." 
time="2017-05-05T20:24:45Z" level=info msg="\tISP: Digital Ocean" 
time="2017-05-05T20:24:45Z" level=info msg="\tIP: 207.154.223.205" 
time="2017-05-05T20:24:45Z" level=info msg="\tCountry: Germany" 
time="2017-05-05T20:24:45Z" level=info msg="\tCity: Frankfurt am Main" 
time="2017-05-05T20:24:45Z" level=info msg="\tZIP: 60438" 
time="2017-05-05T20:24:45Z" level=info msg="\tRegion: HE" 
time="2017-05-05T20:24:45Z" level=info msg="\tLat: 50.1167" 
time="2017-05-05T20:24:45Z" level=info msg="\tLng: 8.6833" 
time="2017-05-05T20:24:45Z" level=info msg="\tTimezone: Europe/Berlin" 
time="2017-05-05T20:24:45Z" level=info msg="Made 1 pools" 
time="2017-05-05T20:24:45Z" level=info msg="Benchmarking drive  /dev/sda"

As you can see the last statement is Benchmarking drive /dev/sda which is my attached volume:

lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda      8:0    0  10G  0 disk 
vda    253:0    0  20G  0 disk 
└─vda1 253:1    0  20G  0 part /

I don't know why it is stuck at this point I have tried numerous ways of getting px-dev working but always encounter this issue.

Does the volume have to be formatted a certain way? Or have some configuration for portworx to use?

I also noticed this in the logs Failed to enable rt scheduler /docker-entry-point.sh: line 749: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied could this be related to the issue?

This was my px-dev startup command:

sudo docker run --restart=always --name px-dev -d --net=host --privileged=true -v /run/docker/plugins:/run/docker/plugins -v /var/lib/osd:/var/lib/osd:shared -v /dev:/dev -v /etc/pwx:/etc/pwx -v /opt/pwx/bin:/export_bin:shared -v /var/run/docker.sock:/var/run/docker.sock -v /var/cores:/var/cores-v /usr/src:/usr/src --ipc=host portworx/px-dev

Replace drive start failed. drive size 32212254720 too big, pool size 1073741824

I'm trying to replace a 1GB drive by a 30GB drive but the following error message is displayed. Why? What I need to do to before to accomplish this?

root@swarm-manager-001:~# /opt/pwx/bin/pxctl service drive replace --source /dev/xvdg --target /dev/xvdi --operation start
Replace drive start failed. drive size 32212254720 too big, pool size 1073741824

Cluster info

root@swarm-manager-001:~# /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: 621ff80d-c47f-42e0-9ab2-fc67dba0b688
	IP: 10.0.1.101 
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	MEDIUM		raid0		2.0 GiB	225 MiB	Online	a	sa-east-1
	Local Storage Devices: 2 devices
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/xvdg	STORAGE_MEDIUM_MAGNETIC	1.0 GiB		11 Jul 17 13:31 UTC
	0:2	/dev/xvdh	STORAGE_MEDIUM_MAGNETIC	1.0 GiB		11 Jul 17 13:31 UTC
	total			-			2.0 GiB
Cluster Summary
	Cluster ID: socialbase
	IP		ID					Used	Capacity	Status
	10.0.1.101	621ff80d-c47f-42e0-9ab2-fc67dba0b688	225 MiB	2.0 GiB		Online (This node)
	10.0.2.102	31a1ff1f-4e12-4a73-9fae-9009ce855144	225 MiB	2.0 GiB		Online
	10.0.3.103	b6505999-5415-422e-9b89-c9dac8e6bf51	0 B	0 B		Online
Global Storage Pool
	Total Used    	:  451 MiB
	Total Capacity	:  4.0 GiB

oot@swarm-manager-001:~# lsblk 
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   12G  0 disk 
└─xvda1 202:1    0   12G  0 part /
xvdf    202:80   0    1G  0 disk 
└─xvdf1 202:81   0 1023M  0 part 
xvdg    202:96   0    2G  0 disk 
xvdh    202:112  0    2G  0 disk 
xvdi    202:128  0   30G  0 disk 

root@swarm-manager-001:~# docker plugin ls
ID                  NAME                DESCRIPTION                         ENABLED
debb3f661bf6        pxd:latest          Portworx Data Services for Docker   true


root@swarm-manager-001:~# docker plugin inspect pxd:latest
[
    {
        "Config": {
            "Args": {
                "Description": "Portworx Volume Driver Options. Refer http://docs.portworx.com for a full list of supported options",
                "Name": "opts",
                "Settable": [
                    "value"
                ],
                "Value": null
            },
            "Description": "Portworx Data Services for Docker",
            "Documentation": "http://docs.portworx.com/",
            "Entrypoint": [
                "/docker-entry-point.sh",
                "-daemon"
            ],
            "Env": [
                {
                    "Description": "",
                    "Name": "GOTRACEBACK",
                    "Settable": null,
                    "Value": "crash"
                },
                {
                    "Description": "",
                    "Name": "GOMAXPROCS",
                    "Settable": null,
                    "Value": "64"
                },
                {
                    "Description": "",
                    "Name": "PXMOD_SOURCE",
                    "Settable": null,
                    "Value": "/home/px-fuse"
                },
                {
                    "Description": "",
                    "Name": "PXMOD_VERSION",
                    "Settable": null,
                    "Value": "2"
                },
                {
                    "Description": "",
                    "Name": "PX_PLUGIN",
                    "Settable": null,
                    "Value": "true"
                }
            ],
            "Interface": {
                "Socket": "pxd.sock",
                "Types": [
                    "docker.volumedriver/1.0"
                ]
            },
            "IpcHost": false,
            "Linux": {
                "AllowAllDevices": true,
                "Capabilities": [
                    "CAP_IPC_LOCK",
                    "CAP_SYS_ADMIN",
                    "CAP_SYS_MODULE"
                ],
                "Devices": null
            },
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/etc/pwx",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/etc/pwx",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/lib/osd",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/lib/osd",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/export_bin",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/opt/pwx/bin",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/run/docker.sock",
                    "Name": "",
                    "Options": [
                        "shared",
                        "bind"
                    ],
                    "Settable": null,
                    "Source": "/var/run/docker.sock",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/lib/modules",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/lib/modules",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/usr/src",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/usr/src",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/cores",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/cores",
                    "Type": "bind"
                }
            ],
            "Network": {
                "Type": "host"
            },
            "PidHost": false,
            "PropagatedMount": "/var/lib/osd",
            "User": {},
            "WorkDir": "",
            "rootfs": {
                "diff_ids": [
                    "sha256:d9519377a52707e2347220a92802441842ad70c6936798d20f60d7c4997ce3a1"
                ],
                "type": "layers"
            }
        },
        "Enabled": true,
        "Id": "debb3f661bf6dbaf494ff474e600a721535ad4750ebf1546b1b544c5cffed5cb",
        "Name": "pxd:latest",
        "PluginReference": "docker.io/portworx/px:latest",
        "Settings": {
            "Args": [
                "-k",
                "consul:http://myconsul:8500",
                "-c",
                "socialbase",
                "-s",
                "/dev/xvdg",
                "-s",
                "/dev/xvdh"
            ],
            "Devices": [],
            "Env": [
                "GOTRACEBACK=crash",
                "GOMAXPROCS=64",
                "PXMOD_SOURCE=/home/px-fuse",
                "PXMOD_VERSION=2",
                "PX_PLUGIN=true"
            ],
            "Mounts": [
                {
                    "Description": "",
                    "Destination": "/dev",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/dev",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/etc/pwx",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/etc/pwx",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/lib/osd",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/lib/osd",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/export_bin",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/opt/pwx/bin",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/run/docker.sock",
                    "Name": "",
                    "Options": [
                        "shared",
                        "bind"
                    ],
                    "Settable": null,
                    "Source": "/var/run/docker.sock",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/lib/modules",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/lib/modules",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/usr/src",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/usr/src",
                    "Type": "bind"
                },
                {
                    "Description": "",
                    "Destination": "/var/cores",
                    "Name": "",
                    "Options": [
                        "rshared",
                        "rbind",
                        "rw"
                    ],
                    "Settable": null,
                    "Source": "/var/cores",
                    "Type": "bind"
                }
            ]
        }
    }
]

How does PX format and partition

I am trying to use px-dev to pick up a device (/dev/ram0) for px-dev use, but the issue I keep running into is the following error when I start PX using a config.json

time="2017-07-31T22:23:31Z" level=info msg="PX cluster manager running." 
time="2017-07-31T22:23:31Z" level=info msg="Starting server on port: 9001" 
time="2017-07-31T22:23:31Z" level=info msg="PX API server running on port 9001." 
time="2017-07-31T22:23:31Z" level=info msg="Adding cluster event listener: PX Storage Service" 
time="2017-07-31T22:23:31Z" level=info msg="Adding cluster event listener: Scheduler" 
time="2017-07-31T22:23:31Z" level=info msg="Cluster manager starting..." 
time="2017-07-31T22:23:31Z" level=info msg="LoggingURL during initializing a new cluster.%s  " 
time="2017-07-31T22:23:31Z" level=info msg="Cluster state is OK... Joining the cluster." 
time="2017-07-31T22:23:31Z" level=info msg="Node 0ad91d4e-10g4-46c9-8152-c9c95bb492b4 joining cluster..." 
time="2017-07-31T22:23:31Z" level=info msg="Cluster ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5" 
time="2017-07-31T22:23:31Z" level=info msg="Node Mgmt IP: 192.168.x.x" 
time="2017-07-31T22:23:31Z" level=info msg="Node Data IP: 192.168.x.x" 
time="2017-07-31T22:23:31Z" level=info msg="Cluster Logging URL : " 
time="2017-07-31T22:23:41Z" level=warning msg="Error while getting service provider information: Get http://ip-api.com/json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" 
time="2017-07-31T22:23:44Z" level=info msg="Made 1 pools" 
time="2017-07-31T22:23:44Z" level=error msg="hal.Renit() Failed with error: Failed to format [-f --nodiscard /dev/ram0]: '/dev/ram0' is too small to make a usable filesystem\nMinimum size for each btrfs device is 41943040.\nbtrfs-progs v4.4\nSee http://btrfs.wiki.kernel.org for more information.\n\n (exit status 1)" 
time="2017-07-31T22:23:44Z" level=info Error=<nil> Function=nodeMap.Add MID=0ad91d4e-10f5-46c9-8152-c9c95bb492b4 NID=14 Status=InitializationError Version= 
time="2017-07-31T22:23:44Z" level=warning msg="Failed to initialize Init PX Storage Service: Storage failed initialization" 
time="2017-07-31T22:23:44Z" level=info msg="Cleanup Init services" 
time="2017-07-31T22:23:44Z" level=warning msg="Cleanup Init for service PX Storage Service." 
time="2017-07-31T22:23:44Z" level=info msg="Cleanup Init for Storage provider PXD" 
time="2017-07-31T22:23:44Z" level=warning msg="Cleanup Init for service Scheduler." 
time="2017-07-31T22:23:44Z" level=info msg="Cleanup Initializing node and joining the cluster 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5..." 
time="2017-07-31T22:23:44Z" level=error msg="Failed to initialize node in cluster. Storage failed initialization" 
time="2017-07-31T22:23:44Z" level=error msg="Cluster Manager Failure: Storage failed initialization" 
time="2017-07-31T22:23:45Z" level=error msg="HTTP error: Unavailable" status=500 
time="2017-07-31T22:23:45Z" level=error msg="Error while calling home: HTTP-500: Unavailable\n" 

I am using ubuntu 16.04 and using the following modprobe command to create a ram block device. I am doing this within virtbox on my local mac, so not lots of memory to allocate to the ram drive

#modprobe brd rd_size=16384

#rmmod brd removes the device successfully as well

this seems to create /dev/ram0, and I am defining that in config.json to have px pick up the device and begin to prepare to use it. However the logs above seems to indicate its having an issue perhaps with the way its partitioning and formatting the device.

thnx for any advice or help you can provide

Kubernetes install option

px-lite had Kubernetes install readme, would be nice to have the same section here as well

Document where and how to provide kubernetes credentials

I'm using kubernetes bootstrapped via tectonic-installer with RBAC enabled. I submitted a kubernetes.yaml/admin.conf likewise:

apiVersion: v1
kind: Config
clusters:
- cluster:
    api-version: v1
    server: https://controller.my-kubenetes.dev:443
    certificate-authority: /etc/kubernetes/ca.crt
preferences:
  colors: true

I also volume-mounted a working kubeconfig to /root/.kube/config. The following error persists:

time="2017-05-23T08:59:37Z" level=error msg="Could not initialize scheduler hooks for kubernetes: the server has asked for the client to provide credentials (get nodes)"

Maven build fails to checkout to local repository on volume mounted with px driver

using Jenkins master/slave docker containers and Rancher as the orchestration platform, I've setup jenkins-slave instance to mount a m2-repository volume created with pxctl command line tool.

When running a maven build, the process starts to checkout project dependencies to local ~/.m2 repository which is a volume manages by px.

The process creates required directories for first dependency and a lastUpdated file with size 0 bytes, all owned by root, yet the process runs under jenkins-slave user. The build then fails with IOException.

I've tried the same setup under no volume and under convoy volume with drivermapper driver and have no issues there. The maven build checks out all dependencies and stores them on ~/.m2 directory with correct permissions.

attach non-formated volume

pxctl v c test -r 3 --fs none docker run --volume-driver pxd -it --rm -v test:/dev/blockdevice ubuntu

error
Error response from daemon: error while mounting volume '/': Volume (Name: test Id: 99558319756929467 Path: /var/lib/osd/mounts/test) mount failed. Volume does not have a filesystem.

How I can use a block device directly from portworx inside a docker? If I attach manually before run docker, it works but it's not the idea
pxctl host attach test docker run -it --rm --device /dev/pxd/pxd<number>:/dev/blockdevice ubuntu <now the blockdevice it is accesible and works>

kubernetes native PV support

kubernetes native PV support would be a very big advantage

Failed with error: Scan found old metadata drives

Reinitialize the node and the following error occurs：

time="2018-05-17T09:18:16Z" level=error msg="hal.Reinit() Failed with error: Scan found old metadata drives"
time="2018-05-17T09:18:16Z" level=info Error= Function=nodeMap.Add MID=79de5140-abec-4b60-9af3-bb4554f25bf8 NID=15 Status=InitializationError Version=
time="2018-05-17T09:18:16Z" level=warning msg="Failed to initialize Init PX Storage Service: Storage failed initialization"
time="2018-05-17T09:18:16Z" level=info msg="Cleanup Init services"
time="2018-05-17T09:18:16Z" level=warning msg="Cleanup Init for service Scheduler."
time="2018-05-17T09:18:16Z" level=info msg="Cleanup Initializing node and joining the cluster 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5..."
time="2018-05-17T09:18:16Z" level=warning msg="Cleanup Init for service PX Storage Service."
time="2018-05-17T09:18:16Z" level=info msg="Cleanup Init for Storage provider PXD"
time="2018-05-17T09:18:16Z" level=error msg="Failed to initialize node in cluster. Storage failed initialization"
time="2018-05-17T09:18:16Z" level=error msg="Cluster Manager Failure: Storage failed initialization"
time="2018-05-17T09:18:16Z" level=warning msg="Removing /etc/pwx/.private.json"
time="2018-05-17T09:18:16Z" level=info msg="Forcing the rescan of Docker drivers"
time="2018-05-17T09:18:16Z" level=error msg="Error while calling home: HTTP-500: Unavailable****

[CoreOS] Failed to init driver: px/flexvolume

Hi,

I wanted to take a look into px-dev and could set it up on my dev-master node but the other node is failing.

Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: E0131 11:22:37.679792   10550 flexvolume_util.go:98] Failed to init driver: px/flexvolume, error: exit status 1
Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: E0131 11:22:37.679826   10550 flexvolume_util.go:79] Failed to unmarshal output for command: init, output: 2017-01-31T11:22:37Z NONE  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 0.0.0.0:9005: getsockopt: connection refused"; Reconnecting to {0.0.0.0:9005 <nil>}
Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: rpc error: code = 14 desc = grpc: the connection is unavailable
Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: , error: invalid character '-' after top-level value
Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: E0131 11:22:37.679851   10550 plugins.go:339] Failed to load volume plugin &{px/flexvolume /etc/kubernetes/kubelet-plugins/volume/exec/px~flexvolume %!s(*kubelet.kubeletVolumeHost=&{0xc4204da000 {{1 0} map[kubernetes.io/downward-api:0xc4202f6610 kubernetes.io/empty-dir:0xc4202f60e0 kubernetes.io/secret:0xc4202f6310 kubernetes.io/nfs:0xc420c7a190 kubernetes.io/flocker:0xc4202f6630 kubernetes.io/fc:0xc420cdf300 kubernetes.io/git-repo:0xc4202f62f0 kubernetes.io/host-path:0xc420c7a140 kubernetes.io/iscsi:0xc420cdf280 kubernetes.io/glusterfs:0xc420cdf2a0 kubernetes.io/rbd:0xc420cdf2c0 kubernetes.io/cinder:0xc420cdf2e0 kubernetes.io/aws-ebs:0xc4202f6010 kubernetes.io/gce-pd:0xc4202f62e0 kubernetes.io/quobyte:0xc4202f6320 kubernetes.io/cephfs:0xc4202f65d0]}})}, error: invalid character '-' after top-level value
Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: Error: failed to run Kubelet: failed to create kubelet: Could not initialize volume plugins for KubeletVolumePluginMgr: invalid character '-' after top-level value
Jan 31 11:22:37 a-10-10-23-86 kubelet-wrapper[10550]: failed to run Kubelet: failed to create kubelet: Could not initialize volume plugins for KubeletVolumePluginMgr: invalid character '-' after top-level value

I'm using the same config as on my master node with a loopback device for testing purposes.

PX Container is running and has no error's in the logs.

sudo /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: 15833480-8141-457d-8fda-a5a6023fc192
	IP: 10.10.23.86
 	Local Storage Pool: 1 pool
	Pool	IO_Priority	Size	Used	Status	Zone	Region
	0	MEDIUM		37 GiB	2.0 GiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/loop0	STORAGE_MEDIUM_MAGNETIC	37 GiB		31 Jan 17 11:31 UTC
	total			-			37 GiB
Cluster Summary
	Cluster ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a7
	Node IP: 10.10.23.86 - Capacity: 2.0 GiB/37 GiB Online (This node)
	Node IP: 10.10.23.10 - Capacity: 2.0 GiB/40 GiB Online
Global Storage Pool
	Total Used    	:  4.0 GiB
	Total Capacity	:  77 GiB

Kubelet is v1.5.1_coreos.0

Edit:
I think the key here is the connection refused on :9005

If I try to run the flexvolume init manually I get the same message:

$ /etc/kubernetes/kubelet-plugins/volume/exec/px~flexvolume/flexvolume init
2017-01-31T12:10:51Z NONE  grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp 0.0.0.0:9005: getsockopt: connection refused"; Reconnecting to {0.0.0.0:9005 <nil>}
rpc error: code = 14 desc = grpc: the connection is unavailable

Kubernetes ReadWriteMany mode problems

Hi there

I'm running into issues when I delete a pod that uses a ReadWriteMany volume claim with px-dev. Nowadays, pods are mostly scheduled using Deployment, which means that a new pod is scheduled as soon as the the previous one gets deleted. The pod using the RWM volume can't get killed and is stuck in the Terminating state forever, while the new pod successfully binds to the same volume and gets Ready immediately. At the same time the following warnings start to appear and continue until the persistent volume (not the claim) gets deleted:

time="2017-04-08T08:22:57Z" level=warning msg="Volume (Name: pvc-0567564a-1c33-11e7-b46b-de2b4425e003 Id: 808123070627826950 Path: /var/lib/kubelet/pods/089721e3-1c33-11e7-b46b-de2b4425e003/volumes/kubernetes.io~portworx-volume/pvc-0567564a-1c33-11e7-b46b-de2b4425e003) unmount failed with error: Mountpath is not mounted" AbortOnError=false BackgroundProcessing=false Driver=pxd Error="Mountpath is not mounted" Format=FS_TYPE_EXT4 Function=Unmount ID=808123070627826950 State=VOLUME_STATE_ATTACHED Version=1570545 

Please note that this does not happen with ReadWriteOnce volume claims. Force deleting the stuck pod using kubectl delete pod <POD> --grace-period=0 --force has no impact, px (or kubernetes itself) will try to unmount the volume forever. This all happens on Kubernetes 1.6.

Thanks for your great work. I'm currently writing a guide where I will most probably mention portworx as the first choice for getting distributed persistent storage in a small-scale setup on virtually any cloud provider.

PX-Dev & Consul: Test connection from KVDB failed

System:

3 Node Swarm Cluster
Ubuntu 16 .04
Docker 1.13

Steps performed:

docker network create --driver overlay consul --attachable

consul.yaml

version: '3'

services:

  consul1:
    image: "progrium/consul:latest"
    hostname: "consul1"
    command: "-server -bootstrap-expect 3 -ui-dir /ui"
    networks:
      - consul

  consul2:
    image: "progrium/consul:latest"
    hostname: "consul2"
    command: "-server -join consul1"
    depends_on:
      - consul1
    networks:
      - consul

  consul3:
    image: "progrium/consul:latest"
    hostname: "consul3"
    command: "-server -join consul1"
    depends_on:
      - consul1
    networks:
      - consul

networks:
  consul:
    external: true

docker stack deploy --compose-file consul.yaml consul

config.json:

{
  "clusterid": "5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5",
  "kvdb": [
              "consul:http://consul1:8500",
              "consul:http://consul2:8500",
              "consul:http://consul3:8500"
         ],
  "storage": {
    "devices": [
      "/dev/vdb"
    ]
  }
}

PX-Dev start:

docker run --restart=always --name px -d --net=consul     \
                 --privileged=true                             \
                 -v /run/docker/plugins:/run/docker/plugins    \
                 -v /var/lib/osd:/var/lib/osd           \
                 -v /dev:/dev                                  \
                 -v /etc/pwx:/etc/pwx                          \
                 -v /opt/pwx/bin:/export_bin            \
                 -v /var/run/docker.sock:/var/run/docker.sock  \
                 -v /var/cores:/var/cores                      \
                 -v /usr/src:/usr/src                          \
                 -v /lib/modules:/lib/modules                  \
                 --ipc=host                                    \
                portworx/px-dev

Verify connectivity:

root@sw1:~# docker exec -it px-dev curl http://consul1:8500
<a href="/ui/">Moved Permanently</a>.

When executing pxctl:

root@sw1:~# /opt/pwx/bin/pxctl help
WARN[0000] Test connection from KVDB failed: Get http://consul1:8500/v1/kv/pwx/5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5/testConnection?consistent=: dial tcp: lookup consul1 on 10.1.94.8:53: no such host
WARN[0001] Test connection from KVDB failed: Get http://consul1:8500/v1/kv/pwx/5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb61a5/testConnection?consistent=: dial tcp: lookup consul1 on 10.1.94.8:53: no such host
Invalid PX Configuration: Configuration check failed: Failed to test connection with KVDB, tried 2
root@sw1:~#

UPDATE:

After Publishing a Consul Port (config.json http://127.0.0.1:8500) and using --net host KVDB connection error disappeard, however i run into a new issue:

root@sw1:~# pxctl status
PX is not running on this host

logs from px-dev container:

time="2017-02-06T12:25:37Z" level=error msg="This node is already initialized but could not be found in the cluster map."
time="2017-02-06T12:25:37Z" level=error msg="Failed to initialize node in cluster. Node is already initialized but not found in the cluster."
time="2017-02-06T12:25:37Z" level=error msg="Could not start cluster manager because of: Node is already initialized but not found in the cluster."
px daemon exited with code: 1

19483
trying to kill px-storage...

2017-02-06 12:25:37,091 INFO exited: pxdaemon (exit status 1; not expected)

portworx on rancher (PX is not running on this host)

portworx service comes up without any problems however pxctl status is reporting that service is not running.

Not sure how to debug this.

:~$ /opt/pwx/bin/pxctl status
PX is not running on this host

Start/Stop systemd scripts for portworx standalone setup

I have successfully set up px-enterprise using below link

https://docs.portworx.com/scheduler/docker/install-standalone.html

Now I looking for systemd start/stop script for this setup. Does anyone try the similar option given in below link?

https://docs.portworx.com/scheduler/docker/install-standalone.html

Multiple errors on dmesg and inestability

System
Ubuntu 16.04
Docker standalone 17.06
3 nodes(Vmware ESXI VM's running portworx)
all the 3 nodes exactly same OS/config

portworx version pxctl version 1.2.11.10-421c67f (OCI)

I got multiple errors on dmesg, one node systemctl portworx goes down(need to reboot to get it working again) and other simply the volume get stucked some seconds so the container mounting the volume get freezed.

I detected that the hosts were the prometheus is running is the node that the OCI service goes down.
(I have a prometheus running and saving data to a portworx volume)
the ID of all the pxd volumes mencioned in the log are
prometheus-data
prometheus-conf
grafana-data
grafana-conf

`
[07.01.18 12:45]
5329.410629] Buffer I/O error on dev pxd/pxd417292423831969158 logical block 262128 async page read

[ 5329.416994] Buffer I/O error on dev pxd/pxd417292423831969158 logical block 1 async page read
[ 5331.303858] EXT4-fs (pxd!pxd417292423831969158): mounted filesystem with ordered data mode. Opts: discard
[ 5334.163731] Buffer I/O error on dev pxd/pxd696187657595542787 logical block 262128 async page read
[ 5334.167582] Buffer I/O error on dev pxd/pxd696187657595542787 logical block 1 async page read
[ 5335.142935] EXT4-fs (pxd!pxd696187657595542787): mounted filesystem with ordered data mode. Opts: discard
[ 5339.061701] Buffer I/O error on dev pxd/pxd417292423831969158 logical block 262128 async page read
[ 5339.065620] Buffer I/O error on dev pxd/pxd417292423831969158 logical block 1 async page read
[ 5339.844751] EXT4-fs (pxd!pxd417292423831969158): mounted filesystem with ordered data mode. Opts: discard
[ 5340.559824] Buffer I/O error on dev pxd/pxd696187657595542787 logical block 262128 async page read
[ 5340.564666] Buffer I/O error on dev pxd/pxd696187657595542787 logical block 1 async page read
[ 5341.326908] EXT4-fs (pxd!pxd696187657595542787): mounted filesystem with ordered data mode. Opts: discard
`

[ 1194.869903] ------------[ cut here ]------------ [ 1194.869916] WARNING: CPU: 24 PID: 10756 at /build/linux-hwe-edge-hwM8VI/linux-hwe-edge-4.13.0/block/blk-core.c:172 blk_update_request+0x257/0x2f0 [ 1194.869918] Modules linked in: xt_set ip_set_hash_net ip_set iptable_raw xt_nat xt_mark xfrm6_mode_tunnel xfrm4_mode_tunnel veth esp4 ansi_cprng ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc px(OE) aufs vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache vmw_vsock_vmci_transport vsock sb_edac intel_rapl_perf ppdev joydev input_leds vmw_balloon serio_raw i2c_piix4 shpchp vmw_vmci parport_pc parport nfit mac_hid ib_iser sunrpc rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy [ 1194.869967] async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel vmwgfx pcbc ttm drm_kms_helper aesni_intel syscopyarea aes_x86_64 sysfillrect mptspi crypto_simd glue_helper isci mpt3sas cryptd sysimgblt mptscsih psmouse fb_sys_fops libsas ahci raid_class mptbase drm vmxnet3 scsi_transport_sas libahci scsi_transport_spi pata_acpi [ 1194.870004] CPU: 24 PID: 10756 Comm: px-storage Tainted: G OE 4.13.0-19-generic #22~16.04.1-Ubuntu [ 1194.870006] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 [ 1194.870009] task: ffff8d31efa245c0 task.stack: ffffb023479e0000 [ 1194.870013] RIP: 0010:blk_update_request+0x257/0x2f0 [ 1194.870015] RSP: 0018:ffffb023479e3c90 EFLAGS: 00010282 [ 1194.870017] RAX: 00000000000000fe RBX: 00000000000000fe RCX: 0000000000000000 [ 1194.870019] RDX: 0000000000001000 RSI: 00000000000000fe RDI: ffff8d3222fc87b0 [ 1194.870021] RBP: ffffb023479e3cc0 R08: ffff8d3203236458 R09: ffff8d32294bc268 [ 1194.870023] R10: ffffb023479e3dd8 R11: ffff8d31efa245c0 R12: 0000000000001000 [ 1194.870025] R13: ffff8d3222fc87b0 R14: 0000000000000000 R15: ffff8d3203236448 [ 1194.870028] FS: 00007f96a59e2700(0000) GS:ffff8d322f400000(0000) knlGS:0000000000000000 [ 1194.870030] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1194.870032] CR2: 000000c41ff52bfa CR3: 00000003f77b0000 CR4: 00000000000406e0 [ 1194.870094] Call Trace: [ 1194.870114] ? pxd_process_write_reply_q+0x30/0x30 [px] [ 1194.870118] blk_update_bidi_request+0x21/0x90 [ 1194.870122] ? pxd_process_write_reply_q+0x30/0x30 [px] [ 1194.870125] blk_end_bidi_request+0x31/0x80 [ 1194.870129] ? pxd_process_write_reply_q+0x30/0x30 [px] [ 1194.870132] blk_end_request+0x22/0x40 [ 1194.870135] pxd_process_write_reply_q+0x1f/0x30 [px] [ 1194.870139] pxd_process_read_reply_q+0xe/0x10 [px] [ 1194.870143] request_end+0xc8/0x120 [px] [ 1194.870146] fuse_dev_write_iter+0x1a1/0x490 [px] [ 1194.870153] new_sync_write+0xdf/0x130 [ 1194.870157] __vfs_write+0x26/0x40 [ 1194.870161] vfs_write+0xb8/0x1b0 [ 1194.870163] SyS_write+0x55/0xc0 [ 1194.870186] entry_SYSCALL_64_fastpath+0x1e/0xa9 [ 1194.870188] RIP: 0033:0x7f96b0dbd4bd [ 1194.870189] RSP: 002b:00007f96a59e1bf0 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 [ 1194.870191] RAX: ffffffffffffffda RBX: 000000003730a000 RCX: 00007f96b0dbd4bd [ 1194.870192] RDX: 0000000000000010 RSI: 00007f96a59e1c00 RDI: 00000000000000ca [ 1194.870193] RBP: 0000000000004650 R08: 00007f96a59e2700 R09: 0000000000000052 [ 1194.870195] R10: 0000000000000000 R11: 0000000000000293 R12: 00007f966d85b400 [ 1194.870196] R13: 8e38e38e38e38e39 R14: ffffffffffffffff R15: 0000000000000000 [ 1194.870198] Code: 8b 49 08 2b 48 34 39 cb 0f 47 d9 41 89 5d 58 4c 89 ef e8 4d 70 00 00 48 83 c4 08 b8 01 00 00 00 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> ff 41 be fb ff ff ff e9 de fd ff ff 41 0f b6 45 44 83 e8 20 [ 1194.870219] ---[ end trace cc4cefc0df2568d5 ]---

Resize existing portworx volume with px-dev pxctl

Does anyone know how to resize an existing volume with pxctl for an existing volume?

I have a volume with the default 1G size, which is too small and need to increase it to 10G.

I have found a link to Portworx documentation that refers to a "volume resize" option. But this does not work - or is not available in the version of pxctl that I have.

I get this error:
No help topic for 'resize'

Version: 1.1.6-693c3b5 - running in Docker containers launched from Rancher.

Error InvalidVolume.ZoneMismatch in scalingUP

I've setup a statefull ami accordig to this guide. In my setup, I've chosen to not use a systemd service startup script, instead, I'm just using --restart=unless-stopped from docker run parameter.

I've created 3 vols in 3 different AZs: sa-east-1a, sa-east-b and sa-east-1c, each one with 30GB:

To create the px container in my statefull ami, I've did the bellow steps:

mkdir -p /etc/pwx
mkdir -p /opt/pwx/bin
mkdir -p /var/lib/osd
mkdir -p /var/cores

docker pull portworx/px-dev:latest

# The bellow command DOESN'T starts the container. It just creates it.
/usr/bin/docker create --net=host --privileged=true     \
      --restart=unless-stopped                       \
      -v /run/docker/plugins:/run/docker/plugins     \
      -v /var/lib/osd:/var/lib/osd:shared            \
      -v /dev:/dev                                   \
      -v /etc/pwx:/etc/pwx                           \
      -v /opt/pwx/bin:/export_bin:shared             \
      -v /var/run/docker.sock:/var/run/docker.sock   \
      -v /var/cores:/var/cores                       \
      -v /usr/src:/usr/src                           \
      --name=portworx \
      portworx/px-dev -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac

After that, I've created the ami and launched a new instance from it with this userdata:

#cloud-config
portworx:
  config:
    storage:
      devices:
        - vol-0842059ddbdbd9d7f
        - vol-058d9c98020c8d4c9
        - vol-002334b285798b4ac
      max-count: 10

runcmd:
 - [ /usr/bin/git, clone, "[email protected]:myenv/repo.git", /opt/docker/ ]
 - [ /bin/bash, -c, "cd /opt/docker/ ; git pull" ]
 - [ /opt/docker/scripts/userdata/userdata.sh, userdata_infra.conf ]

After instance is running, pxctl status shows this:

PX is not running on this host

Checking the AWS console, I've noticed that portworx created a lot of 30GB disks in all 3 zones!

This is the container log. It seems that it entered in a looping state:

Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac
Mon Jul 17 20:19:36 UTC 2017 : Running on Linux ip-10-0-1-73 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
checking /hostusr/src/kernels/4.4.0-83-generic
checking /hostusr/src/linux-headers-4.4.0-83-generic
checking /usr/src/kernels/4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Flushing buffer-cache..
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Creating px fs...
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Using storage device: vol-058d9c98020c8d4c9
Using storage device: vol-002334b285798b4ac
Fingerprints values:
HOSTNAME={"type": 1, "value": "ip-10-0-1-73"}
DISK={ "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }
NETWORK={ "type": 7, "value": "" }
NETWORK={ "type": 8, "value": "" }

FP_FINGERPRINT="Fingerprint": [{"type": 1, "value": "ip-10-0-1-73"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz" }, { "type": 5, "value": "3852904 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }]


FP_CONFIG={ "token": "", "create_profile": true, "profile_name": "ip-10-0-1-73-NoqPxviZ", "Fingerprint": [{"type": 1, "value": "ip-10-0-1-73"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz" }, { "type": 5, "value": "3852904 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }] }
To use the same hardware configuration on an identical node, 		please use hardware profile ip-10-0-1-73-NoqPxviZ.
example:
    curl -fsSL http://lighthouse.portworx.com/px_bootstrap.sh > px_bootstrap.sh;  chmod u+x px_bootstrap.sh;  PWX_TOKEN= PWX_CLUSTERNAME=socialbase 		bootstrap.sh -p ip-10-0-1-73-NoqPxviZ
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Failed to enable rt scheduler
Checking sysfs mount...
Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac
Mon Jul 17 20:20:03 UTC 2017 : Running on Linux ip-10-0-1-73 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
PXD version:  33cafba6c39c862340a8c30b2677849f67bd2d6a
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Using storage device: vol-058d9c98020c8d4c9
Using storage device: vol-002334b285798b4ac
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Failed to enable rt scheduler
Checking sysfs mount...
2017-07-17 20:20:04,943 CRIT Supervisor running as root (no user in config file)
2017-07-17 20:20:04,947 INFO supervisord started with pid 1
2017-07-17 20:20:05,949 INFO spawned: 'relayd' with pid 75
2017-07-17 20:20:05,950 INFO spawned: 'lttng' with pid 76
2017-07-17 20:20:05,952 INFO spawned: 'exec' with pid 77
2017-07-17 20:20:05,953 INFO spawned: 'pxdaemon' with pid 78
2017-07-17 20:20:05,954 INFO spawned: 'px-ns' with pid 79
2017-07-17 20:20:05,956 INFO spawned: 'px_event_listener' with pid 80
PXPROCS: lttng not started yet...sleeping 

Mon Jul 17 20:20:06 UTC 2017 cleaning up old traces
Mon Jul 17 20:20:06 UTC 2017 done cleaning up old traces
time="2017-07-17T20:20:07Z" level=info msg="px-ns Starting.." 
2017-07-17 20:20:07,139 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,139 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: px-ns entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
NS client starting fuse module
Starting NS server
PXPROCS: lttng not started yet...sleeping 

PXPROCS: lttng not started yet...sleeping 

PXPROCS: lttng not started yet...sleeping 

Spawning a session daemon
Session pxd created.
Traces will be written in net://localhost
Live timer set to 2000000 usec
Session pxd set to shm_path: /var/lib/osd/lttng/pxd-20170717-202016.
UST channel pxd_channel enabled for session pxd
All UST events are enabled in channel pxd_channel
Tracing started for session pxd
PXPROCS: Started px-storage with pid 152
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
PXPROCS: Started px with pid 163
PXPROCS: Started watchdog with pid 164
2017-07-17_20:20:21: PX-Watchdog: Starting watcher
2017-07-17_20:20:22: PX-Watchdog: Waiting for px process to start
root       163    78  0 20:20 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:20:23: PX-Watchdog: (pid 163): Begin monitoring
time="2017-07-17T20:20:28Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:20:28Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:20:29Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 1b35b6ce-8d86-41dc-9636-373f359a3af9). Trying attachSet..." 
time="2017-07-17T20:20:29Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:20:29Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:20:29Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:20:29Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:20:29Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:20:29Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:20:29Z" level=info msg="New NodeID is db74ffaa-5541-4d7e-a78d-c44f709ecc97" 
time="2017-07-17T20:20:29Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:20:29Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:20:29Z" level=info msg="Node db74ffaa-5541-4d7e-a78d-c44f709ecc97 with Index (184) is Up" 
time="2017-07-17T20:20:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d5a0524e-b2a6-4d98-a0f7-f31bac8b3f78). Trying attachSet..." 
time="2017-07-17T20:20:47Z" level=info msg="Failed to attach volume set  vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..." 
time="2017-07-17T20:20:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d7c87f0b-0833-4d22-a357-001ee84188fc). Trying attachSet..." 
time="2017-07-17T20:20:50Z" level=info msg="Failed to attach volume set  vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..." 
time="2017-07-17T20:20:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 944f071b-d693-4b5a-9f55-8d09132194ec). Trying attachSet..." 
time="2017-07-17T20:20:53Z" level=info msg="Failed to attach volume set  vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..." 
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-0a1c8c9f619baf16e" 
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-00b0fb36e769745ad" 
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-0dc72eda0ec3c725a" 
time="2017-07-17T20:20:57Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

152
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:20:58,301 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:20:59,303 INFO spawned: 'pxdaemon' with pid 214
PXPROCS: Started px-storage with pid 218
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:21:00,696 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 239
PXPROCS: Started watchdog with pid 240
2017-07-17_20:21:02: PX-Watchdog: Starting watcher
2017-07-17_20:21:02: PX-Watchdog: Waiting for px process to start
root       239   214  0 20:21 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:21:02: PX-Watchdog: (pid 239): Begin monitoring
time="2017-07-17T20:21:02Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:21:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:21:03Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 112b36fa-e59e-44f9-83cc-9c76515e5bc7). Trying attachSet..." 
time="2017-07-17T20:21:03Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:21:03Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:21:03Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:21:03Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:21:03Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:21:03Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:21:03Z" level=info msg="New NodeID is f838f2b1-5e9c-45a0-a8ca-ec3d3f24bb00" 
time="2017-07-17T20:21:03Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:21:03Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:21:03Z" level=info msg="Node f838f2b1-5e9c-45a0-a8ca-ec3d3f24bb00 with Index (185) is Up" 
time="2017-07-17T20:21:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d2d87553-9173-4c45-b459-6819b68ead08). Trying attachSet..." 
time="2017-07-17T20:21:17Z" level=info msg="Failed to attach volume set  vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:21Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8633021a-561b-452c-aead-2e824e743193). Trying attachSet..." 
time="2017-07-17T20:21:21Z" level=info msg="Failed to attach volume set  vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:24Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d53417a1-eb9f-42bc-8132-5161445eae3c). Trying attachSet..." 
time="2017-07-17T20:21:24Z" level=info msg="Failed to attach volume set  vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..." 
2017-07-17 20:21:26,693 INFO reaped unknown pid 200
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-00750f1049532859a" 
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-02f7cbe7bcbafa260" 
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-0d9c96794a8ddfd40" 
time="2017-07-17T20:21:27Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

218
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:21:28,702 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:21:29,705 INFO spawned: 'pxdaemon' with pid 323
PXPROCS: Started px-storage with pid 324
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:21:31,346 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 333
PXPROCS: Started watchdog with pid 334
2017-07-17_20:21:32: PX-Watchdog: Starting watcher
2017-07-17_20:21:32: PX-Watchdog: Waiting for px process to start
root       333   323  0 20:21 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:21:32: PX-Watchdog: (pid 333): Begin monitoring
time="2017-07-17T20:21:33Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:21:33Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:21:34Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4996b685-5ba7-4937-9c9d-aff07f2ba1a5). Trying attachSet..." 
time="2017-07-17T20:21:34Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:21:34Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:21:34Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:21:34Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:21:34Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:21:34Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:21:34Z" level=info msg="New NodeID is e3f28af0-422e-4cb3-a4bf-877932e25ebb" 
time="2017-07-17T20:21:34Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:21:34Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:21:34Z" level=info msg="Node e3f28af0-422e-4cb3-a4bf-877932e25ebb with Index (186) is Up" 
time="2017-07-17T20:21:51Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 886f626e-2a8f-43a8-a048-6e5903792217). Trying attachSet..." 
time="2017-07-17T20:21:51Z" level=info msg="Failed to attach volume set  vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:54Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6b193abd-1eb6-4179-af56-c2233c08b060). Trying attachSet..." 
time="2017-07-17T20:21:54Z" level=info msg="Failed to attach volume set  vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..." 
time="2017-07-17T20:21:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: e6841a93-07c8-483f-9439-91a51e1d6532). Trying attachSet..." 
time="2017-07-17T20:21:58Z" level=info msg="Failed to attach volume set  vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-07bef9c07f4ca1bb7" 
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-0c484dcc58221cbee" 
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-0cae8c2f29ba14819" 
time="2017-07-17T20:22:01Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

324
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:22:02,681 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:22:03,683 INFO spawned: 'pxdaemon' with pid 384
PXPROCS: Started px-storage with pid 385
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:22:05,318 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:22:05,381 INFO reaped unknown pid 285
PXPROCS: Started px with pid 394
PXPROCS: Started watchdog with pid 395
2017-07-17_20:22:06: PX-Watchdog: Starting watcher
2017-07-17_20:22:06: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid218-time1500322887
2017-07-17_20:22:06: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:21:27.956992962 +0000 UTC.stack
2017-07-17_20:22:06: PX-Watchdog: Waiting for px process to start
root       394   384  0 20:22 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:22:06: PX-Watchdog: (pid 394): Begin monitoring
time="2017-07-17T20:22:06Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:22:06Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:22:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 04a1c231-6237-43e4-99f0-2a18b14e0409). Trying attachSet..." 
time="2017-07-17T20:22:07Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:22:07Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:22:07Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:22:07Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:22:07Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:22:07Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:22:07Z" level=info msg="New NodeID is 1e53450a-d320-47f3-82e8-e3e38cfad2f6" 
time="2017-07-17T20:22:07Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:22:07Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:22:07Z" level=info msg="Node 1e53450a-d320-47f3-82e8-e3e38cfad2f6 with Index (187) is Up" 
Waiting for state transition to "attached".
time="2017-07-17T20:22:24Z" level=warning msg="Rollback attach volume /dev/xvdf, Error InvalidVolume.ZoneMismatch: The volume 'vol-0754cdc54af687106' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 06831c87-fd63-4363-a7a1-04b4e04021be" 
time="2017-07-17T20:22:24Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0754cdc54af687106' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 06831c87-fd63-4363-a7a1-04b4e04021be). Trying attachSet..." 
time="2017-07-17T20:22:24Z" level=info msg="Failed to attach volume set  vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:27Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-03fb2827619da2903 is already attached to an instance\n\tstatus code: 400, request id: 415096f6-2eb7-49f0-b255-be4807792aa2). Trying attachSet..." 
time="2017-07-17T20:22:28Z" level=info msg="Failed to attach volume set  vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:31Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-03fb2827619da2903 is already attached to an instance\n\tstatus code: 400, request id: 026f0c84-b798-4a6e-884d-20cfbdc2e26e). Trying attachSet..." 
time="2017-07-17T20:22:31Z" level=info msg="Failed to attach volume set  vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..." 
time="2017-07-17T20:22:34Z" level=info msg="Deleted volume vol-0754cdc54af687106" 
time="2017-07-17T20:22:35Z" level=info msg="Deleted volume vol-0e9f2af07873a2459" 
time="2017-07-17T20:22:35Z" level=warning msg="Delete failed for volume vol-03fb2827619da2903, Error VolumeInUse: Volume vol-03fb2827619da2903 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 034beb8e-1b61-459b-8c8b-efb169ff6e46" 
time="2017-07-17T20:22:35Z" level=info msg="Deleted volume vol-03fb2827619da2903" 
time="2017-07-17T20:22:35Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

385
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:22:35,799 INFO reaped unknown pid 376
2017-07-17 20:22:35,833 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:22:36,835 INFO spawned: 'pxdaemon' with pid 451
PXPROCS: Started px-storage with pid 452
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:22:38,470 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 461
PXPROCS: Started watchdog with pid 462
2017-07-17_20:22:39: PX-Watchdog: Starting watcher
2017-07-17_20:22:39: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid324-time1500322921
2017-07-17_20:22:40: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:22:01.687706644 +0000 UTC.stack
2017-07-17_20:22:40: PX-Watchdog: Waiting for px process to start
root       461   451 14 20:22 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:22:40: PX-Watchdog: (pid 461): Begin monitoring
time="2017-07-17T20:22:40Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:22:40Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:22:40Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 70ef01ce-5998-4e22-bb2c-6a42a851e9f4). Trying attachSet..." 
time="2017-07-17T20:22:40Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:22:40Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:22:40Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:22:40Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:22:40Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:22:40Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:22:40Z" level=info msg="New NodeID is aaf41dec-5162-48e0-a638-2090a13836e3" 
time="2017-07-17T20:22:40Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:22:40Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:22:40Z" level=info msg="Node aaf41dec-5162-48e0-a638-2090a13836e3 with Index (188) is Up" 
Waiting for state transition to "attached".
time="2017-07-17T20:22:57Z" level=warning msg="Rollback attach volume /dev/xvdg, Error InvalidVolume.ZoneMismatch: The volume 'vol-05fea0bb3a91338e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a61b135c-31b4-4c28-b5e9-5fce6af98684" 
time="2017-07-17T20:22:57Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-05fea0bb3a91338e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a61b135c-31b4-4c28-b5e9-5fce6af98684). Trying attachSet..." 
time="2017-07-17T20:22:58Z" level=info msg="Failed to attach volume set  vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:01Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-039e4ec32142653e2 is already attached to an instance\n\tstatus code: 400, request id: d6b02a83-9411-4750-8147-61c7cbd014f8). Trying attachSet..." 
time="2017-07-17T20:23:01Z" level=info msg="Failed to attach volume set  vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:05Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-039e4ec32142653e2 is already attached to an instance\n\tstatus code: 400, request id: a894b483-2b4e-4fd5-beda-13f96b58ea91). Trying attachSet..." 
time="2017-07-17T20:23:05Z" level=info msg="Failed to attach volume set  vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:08Z" level=info msg="Deleted volume vol-0c2263b7765a2b50b" 
time="2017-07-17T20:23:09Z" level=info msg="Deleted volume vol-05fea0bb3a91338e3" 
time="2017-07-17T20:23:09Z" level=warning msg="Delete failed for volume vol-039e4ec32142653e2, Error VolumeInUse: Volume vol-039e4ec32142653e2 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: af83fcb4-4f42-4532-b6a7-5dd3cd8e453b" 
time="2017-07-17T20:23:09Z" level=info msg="Deleted volume vol-039e4ec32142653e2" 
time="2017-07-17T20:23:09Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

452
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:23:09,908 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:23:09,931 INFO spawned: 'pxdaemon' with pid 518
2017-07-17 20:23:09,932 INFO reaped unknown pid 442
PXPROCS: Started px-storage with pid 519
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:23:11,565 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 528
PXPROCS: Started watchdog with pid 529
2017-07-17_20:23:12: PX-Watchdog: Starting watcher
2017-07-17_20:23:13: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid385-time1500322955
2017-07-17_20:23:13: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:22:35.164295909 +0000 UTC.stack
2017-07-17_20:23:13: PX-Watchdog: Waiting for px process to start
root       528   518  0 20:23 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:23:13: PX-Watchdog: (pid 528): Begin monitoring
time="2017-07-17T20:23:13Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:23:13Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:23:13Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc68d073-1e09-4809-8d86-3afafba43fc1). Trying attachSet..." 
time="2017-07-17T20:23:13Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:23:13Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:23:13Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:23:13Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:23:13Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:23:13Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:23:13Z" level=info msg="New NodeID is 0d56d20b-4c84-4505-a52e-b7faa353e301" 
time="2017-07-17T20:23:13Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:23:13Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:23:13Z" level=info msg="Node 0d56d20b-4c84-4505-a52e-b7faa353e301 with Index (189) is Up" 
time="2017-07-17T20:23:30Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 1b982094-f18f-454e-8bfd-0e8c410140a9). Trying attachSet..." 
time="2017-07-17T20:23:31Z" level=info msg="Failed to attach volume set  vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:34Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c15f63b7-e38a-4f33-a933-4b44914d3081). Trying attachSet..." 
time="2017-07-17T20:23:34Z" level=info msg="Failed to attach volume set  vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 383087a1-63f7-40c6-903a-daaacf5ad088). Trying attachSet..." 
time="2017-07-17T20:23:38Z" level=info msg="Failed to attach volume set  vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..." 
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-0e0355738c9daf989" 
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-03ce75f45bd0bc5e6" 
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-0b4104817dc18bb25" 
time="2017-07-17T20:23:41Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

519
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:23:42,424 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:23:43,056 INFO spawned: 'pxdaemon' with pid 583
2017-07-17 20:23:43,057 INFO reaped unknown pid 509
PXPROCS: Started px-storage with pid 584
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:23:44,678 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 593
PXPROCS: Started watchdog with pid 594
2017-07-17_20:23:46: PX-Watchdog: Starting watcher
2017-07-17_20:23:46: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid452-time1500322989
2017-07-17_20:23:46: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:23:09.155729831 +0000 UTC.stack
2017-07-17_20:23:46: PX-Watchdog: Waiting for px process to start
root       593   583  0 20:23 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:23:46: PX-Watchdog: (pid 593): Begin monitoring
time="2017-07-17T20:23:46Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:23:46Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:23:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 205a8bb7-2ea4-42ad-8b07-c83138bcb455). Trying attachSet..." 
time="2017-07-17T20:23:47Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:23:47Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:23:47Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:23:47Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:23:47Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:23:47Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:23:47Z" level=info msg="New NodeID is 43b0f079-e184-47fa-a57e-3ddcefacc686" 
time="2017-07-17T20:23:47Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:23:47Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:23:47Z" level=info msg="Node 43b0f079-e184-47fa-a57e-3ddcefacc686 with Index (190) is Up" 
time="2017-07-17T20:24:00Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d6383b44-e221-4625-88ae-0f4d675ce49e). Trying attachSet..." 
time="2017-07-17T20:24:01Z" level=info msg="Failed to attach volume set  vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..." 
time="2017-07-17T20:24:04Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b7bf557a-6236-4aa3-9e8e-1aaecf7be7cc). Trying attachSet..." 
time="2017-07-17T20:24:04Z" level=info msg="Failed to attach volume set  vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..." 
time="2017-07-17T20:24:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ff8f6a0-5333-4023-894e-cd9541c4cc6d). Trying attachSet..." 
time="2017-07-17T20:24:07Z" level=info msg="Failed to attach volume set  vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..." 
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-0602b317886ec4938" 
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-021ace0cd6005e01d" 
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-073cc4cd01ab0067a" 
time="2017-07-17T20:24:11Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

584
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:24:12,374 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:24:13,377 INFO spawned: 'pxdaemon' with pid 647
PXPROCS: Started px-storage with pid 648
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:24:15,018 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:24:16,116 INFO reaped unknown pid 576
PXPROCS: Started px with pid 657
PXPROCS: Started watchdog with pid 658
2017-07-17_20:24:16: PX-Watchdog: Starting watcher
2017-07-17_20:24:16: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid519-time1500323021
2017-07-17_20:24:16: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:23:41.620964623 +0000 UTC.stack
2017-07-17_20:24:16: PX-Watchdog: Waiting for px process to start
root       657   647  0 20:24 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:24:16: PX-Watchdog: (pid 657): Begin monitoring
time="2017-07-17T20:24:16Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:24:16Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:24:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ff2462f-adad-4d28-a4df-7ebc3abad72f). Trying attachSet..." 
time="2017-07-17T20:24:17Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:24:17Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:24:17Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:24:17Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:24:17Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:24:17Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:24:17Z" level=info msg="New NodeID is 80a52d5c-9f0b-4d91-a55d-0100a0d21543" 
time="2017-07-17T20:24:17Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:24:17Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:24:17Z" level=info msg="Node 80a52d5c-9f0b-4d91-a55d-0100a0d21543 with Index (191) is Up" 
time="2017-07-17T20:24:28Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d18194c7-c97c-4653-b74f-c4c327ddf09a). Trying attachSet..." 
time="2017-07-17T20:24:28Z" level=info msg="Failed to attach volume set  vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..." 
time="2017-07-17T20:24:31Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bc0b728a-09ef-4cac-b2e8-693c940c9969). Trying attachSet..." 
time="2017-07-17T20:24:31Z" level=info msg="Failed to attach volume set  vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..." 
time="2017-07-17T20:24:35Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d926c609-e296-47d4-9b71-ef1a3badae7f). Trying attachSet..." 
time="2017-07-17T20:24:35Z" level=info msg="Failed to attach volume set  vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..." 
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-041b6e029a322b566" 
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-053f068cbfcb41ee8" 
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-047bb73b87e28b7a4" 
time="2017-07-17T20:24:38Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

648
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:24:39,561 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:24:40,563 INFO spawned: 'pxdaemon' with pid 715
PXPROCS: Started px-storage with pid 716
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:24:42,172 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 725
PXPROCS: Started watchdog with pid 726
2017-07-17_20:24:43: PX-Watchdog: Starting watcher
2017-07-17_20:24:43: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid584-time1500323051
2017-07-17_20:24:43: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:24:11.35037889 +0000 UTC.stack
2017-07-17_20:24:43: PX-Watchdog: Waiting for px process to start
root       725   715  0 20:24 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:24:43: PX-Watchdog: (pid 725): Begin monitoring
time="2017-07-17T20:24:43Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:24:43Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:24:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 05156c82-e040-4670-977b-8ac6d28b5b2f). Trying attachSet..." 
time="2017-07-17T20:24:44Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:24:44Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:24:44Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:24:44Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:24:44Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:24:44Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:24:44Z" level=info msg="New NodeID is e7a373e2-8745-4a34-a0b7-5d874ab16358" 
time="2017-07-17T20:24:44Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:24:44Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:24:44Z" level=info msg="Node e7a373e2-8745-4a34-a0b7-5d874ab16358 with Index (192) is Up" 
2017-07-17 20:24:49,228 INFO reaped unknown pid 641
time="2017-07-17T20:24:55Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6cdef988-ff36-4089-91de-0a7d31d36a06). Trying attachSet..." 
time="2017-07-17T20:24:55Z" level=info msg="Failed to attach volume set  vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..." 
time="2017-07-17T20:24:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6fbd4955-6ead-4297-9394-8023998e90a1). Trying attachSet..." 
time="2017-07-17T20:24:58Z" level=info msg="Failed to attach volume set  vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..." 
time="2017-07-17T20:25:02Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6799674c-d090-4f39-b2d4-8d4f26a89424). Trying attachSet..." 
time="2017-07-17T20:25:02Z" level=info msg="Failed to attach volume set  vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..." 
time="2017-07-17T20:25:05Z" level=info msg="Deleted volume vol-0657b54d2782a7bc6" 
time="2017-07-17T20:25:05Z" level=info msg="Deleted volume vol-0bb70306e52984a56" 
time="2017-07-17T20:25:06Z" level=info msg="Deleted volume vol-0d703b168c35f282d" 
time="2017-07-17T20:25:06Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

716
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:25:06,799 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:25:07,802 INFO spawned: 'pxdaemon' with pid 779
PXPROCS: Started px-storage with pid 780
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:25:09,410 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 789
PXPROCS: Started watchdog with pid 790
2017-07-17_20:25:10: PX-Watchdog: Starting watcher
2017-07-17_20:25:10: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid648-time1500323078
2017-07-17_20:25:10: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:24:38.685498093 +0000 UTC.stack
2017-07-17_20:25:10: PX-Watchdog: Waiting for px process to start
root       789   779  8 20:25 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:25:10: PX-Watchdog: (pid 789): Begin monitoring
time="2017-07-17T20:25:11Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:25:11Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:25:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4b606935-7d8b-45ed-b606-404ad255af68). Trying attachSet..." 
time="2017-07-17T20:25:11Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:25:11Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:25:11Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:25:11Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:25:11Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:25:11Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:25:11Z" level=info msg="New NodeID is 10777ebd-50b0-4fa5-be24-fe3f1853350c" 
time="2017-07-17T20:25:11Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:25:11Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:25:11Z" level=info msg="Node 10777ebd-50b0-4fa5-be24-fe3f1853350c with Index (193) is Up" 
2017-07-17 20:25:19,526 INFO reaped unknown pid 705
time="2017-07-17T20:25:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 03968f47-0d51-4711-b4bc-299d89bceadb). Trying attachSet..." 
time="2017-07-17T20:25:22Z" level=info msg="Failed to attach volume set  vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..." 
time="2017-07-17T20:25:25Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c8964c28-6a18-44ad-9d9e-8a981987690a). Trying attachSet..." 
time="2017-07-17T20:25:26Z" level=info msg="Failed to attach volume set  vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..." 
time="2017-07-17T20:25:29Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 33939c5f-7e61-4c5a-9efd-adaa5fb93181). Trying attachSet..." 
time="2017-07-17T20:25:29Z" level=info msg="Failed to attach volume set  vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..." 
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0bc3d3459e618fd8b" 
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0ccd3d4f8966fd354" 
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0f101dfd08e86c2c5" 
time="2017-07-17T20:25:32Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

780
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:25:33,524 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:25:34,527 INFO spawned: 'pxdaemon' with pid 844
PXPROCS: Started px-storage with pid 845
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:25:36,158 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 854
PXPROCS: Started watchdog with pid 855
2017-07-17_20:25:37: PX-Watchdog: Starting watcher
2017-07-17_20:25:37: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid716-time1500323106
2017-07-17_20:25:37: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:25:06.079558835 +0000 UTC.stack
2017-07-17_20:25:37: PX-Watchdog: Waiting for px process to start
root       854   844  0 20:25 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:25:37: PX-Watchdog: (pid 854): Begin monitoring
time="2017-07-17T20:25:37Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:25:37Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:25:38Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 49eb1ddd-35e3-423c-a54d-14821585fa39). Trying attachSet..." 
time="2017-07-17T20:25:38Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:25:38Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:25:38Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:25:38Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:25:38Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:25:38Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:25:38Z" level=info msg="New NodeID is 1cb08f7e-ec86-41de-a2f9-6adc19314511" 
time="2017-07-17T20:25:38Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:25:38Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:25:38Z" level=info msg="Node 1cb08f7e-ec86-41de-a2f9-6adc19314511 with Index (194) is Up" 
2017-07-17 20:25:46,728 INFO reaped unknown pid 773
time="2017-07-17T20:25:52Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4130e767-414f-4566-8a7c-04ad1f787ffb). Trying attachSet..." 
time="2017-07-17T20:25:52Z" level=info msg="Failed to attach volume set  vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..." 
time="2017-07-17T20:25:56Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc67c369-235e-4463-aab4-f8ec7cd0d290). Trying attachSet..." 
time="2017-07-17T20:25:56Z" level=info msg="Failed to attach volume set  vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..." 
time="2017-07-17T20:25:59Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 7d7e1823-c775-44b8-ba7f-e9ff9df45a9f). Trying attachSet..." 
time="2017-07-17T20:25:59Z" level=info msg="Failed to attach volume set  vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..." 
time="2017-07-17T20:26:02Z" level=info msg="Deleted volume vol-01a5dd80d92e4d7b9" 
time="2017-07-17T20:26:02Z" level=info msg="Deleted volume vol-0b4b020d6fa2b965f" 
time="2017-07-17T20:26:03Z" level=info msg="Deleted volume vol-080f3e66e286102cb" 
time="2017-07-17T20:26:03Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

845
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:26:03,930 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:26:04,933 INFO spawned: 'pxdaemon' with pid 910
PXPROCS: Started px-storage with pid 911
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:26:06,583 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 920
PXPROCS: Started watchdog with pid 921
2017-07-17_20:26:07: PX-Watchdog: Starting watcher
2017-07-17_20:26:08: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid780-time1500323132
2017-07-17_20:26:08: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:25:32.845892519 +0000 UTC.stack
2017-07-17_20:26:08: PX-Watchdog: Waiting for px process to start
root       920   910  0 20:26 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:26:08: PX-Watchdog: (pid 920): Begin monitoring
time="2017-07-17T20:26:08Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:26:08Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:26:08Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8da904b6-3771-4a9a-a141-50377f617384). Trying attachSet..." 
time="2017-07-17T20:26:08Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:26:08Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:26:08Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:26:08Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:26:09Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:26:09Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:26:09Z" level=info msg="New NodeID is f081adb9-2e28-4eae-819c-cac8a914ec5a" 
time="2017-07-17T20:26:09Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:26:09Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:26:09Z" level=info msg="Node f081adb9-2e28-4eae-819c-cac8a914ec5a with Index (195) is Up" 
2017-07-17 20:26:13,991 INFO reaped unknown pid 838
Waiting for state transition to "attached".
time="2017-07-17T20:26:22Z" level=warning msg="Rollback attach volume /dev/xvdh, Error InvalidVolume.ZoneMismatch: The volume 'vol-050a22aa7316ebf6c' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0544a301-a544-47a2-8355-51c49af6d571" 
time="2017-07-17T20:26:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-050a22aa7316ebf6c' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0544a301-a544-47a2-8355-51c49af6d571). Trying attachSet..." 
time="2017-07-17T20:26:22Z" level=info msg="Failed to attach volume set  vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..." 
time="2017-07-17T20:26:26Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01005566936b4286a is already attached to an instance\n\tstatus code: 400, request id: f4c9ed99-6e10-4b8c-8816-c9bc7ce24df0). Trying attachSet..." 
time="2017-07-17T20:26:26Z" level=info msg="Failed to attach volume set  vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..." 
time="2017-07-17T20:26:29Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01005566936b4286a is already attached to an instance\n\tstatus code: 400, request id: 019de502-07d1-4eb9-916a-17447ee32a1b). Trying attachSet..." 
time="2017-07-17T20:26:29Z" level=info msg="Failed to attach volume set  vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..." 
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-050a22aa7316ebf6c" 
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-0eb0a3b0133732862" 
time="2017-07-17T20:26:33Z" level=warning msg="Delete failed for volume vol-01005566936b4286a, Error VolumeInUse: Volume vol-01005566936b4286a is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: c3786596-2d41-4cfe-b653-2907144b0cd9" 
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-01005566936b4286a" 
time="2017-07-17T20:26:33Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

911
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:26:34,013 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:26:35,017 INFO spawned: 'pxdaemon' with pid 978
PXPROCS: Started px-storage with pid 979
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:26:36,651 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 988
PXPROCS: Started watchdog with pid 989
2017-07-17_20:26:38: PX-Watchdog: Starting watcher
2017-07-17_20:26:38: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid845-time1500323163
2017-07-17_20:26:38: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:26:03.08213779 +0000 UTC.stack
2017-07-17_20:26:38: PX-Watchdog: Waiting for px process to start
root       988   978  0 20:26 ?        00:00:00 /usr/local/bin/px -daemon
time="2017-07-17T20:26:38Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:26:38Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-17_20:26:38: PX-Watchdog: (pid 988): Begin monitoring
time="2017-07-17T20:26:38Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 042f48e1-4a74-4282-b62c-120f0b505275). Trying attachSet..." 
time="2017-07-17T20:26:38Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:26:38Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:26:38Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:26:38Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:26:38Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:26:38Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:26:38Z" level=info msg="New NodeID is 7fa4353d-bdb0-407e-ad50-61009a93446e" 
time="2017-07-17T20:26:38Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:26:38Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:26:38Z" level=info msg="Node 7fa4353d-bdb0-407e-ad50-61009a93446e with Index (196) is Up" 
2017-07-17 20:26:40,727 INFO reaped unknown pid 902
time="2017-07-17T20:26:49Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 249eedb9-03d6-47bc-90e5-bac4a438d78d). Trying attachSet..." 
time="2017-07-17T20:26:50Z" level=info msg="Failed to attach volume set  vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..." 
time="2017-07-17T20:26:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6732b0b1-4591-4653-b9a7-196cd8427385). Trying attachSet..." 
time="2017-07-17T20:26:53Z" level=info msg="Failed to attach volume set  vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..." 
time="2017-07-17T20:26:56Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 010aa708-329c-4477-91a7-309b61759a88). Trying attachSet..." 
time="2017-07-17T20:26:56Z" level=info msg="Failed to attach volume set  vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..." 
time="2017-07-17T20:26:59Z" level=info msg="Deleted volume vol-099d813e8fd60f787" 
time="2017-07-17T20:26:59Z" level=info msg="Deleted volume vol-0ebc07b08c4cde4d3" 
time="2017-07-17T20:27:00Z" level=info msg="Deleted volume vol-0ee606659f735d922" 
time="2017-07-17T20:27:00Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

979
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:27:01,063 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:02,066 INFO spawned: 'pxdaemon' with pid 1044
PXPROCS: Started px-storage with pid 1045
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:27:03,694 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1054
PXPROCS: Started watchdog with pid 1055
2017-07-17_20:27:05: PX-Watchdog: Starting watcher
2017-07-17_20:27:05: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid911-time1500323193
2017-07-17_20:27:05: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:26:33.397134213 +0000 UTC.stack
2017-07-17_20:27:05: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:27:05Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:27:05Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1054  1044  0 20:27 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:27:05: PX-Watchdog: (pid 1054): Begin monitoring
time="2017-07-17T20:27:05Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: dafee0bb-631e-485b-8a8b-ba2b859a38a8). Trying attachSet..." 
time="2017-07-17T20:27:06Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:27:06Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:27:06Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:27:06Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:27:06Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:27:06Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:27:06Z" level=info msg="New NodeID is 20d763d5-b44b-40fc-83f3-9976aa6090d6" 
time="2017-07-17T20:27:06Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:27:06Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:27:06Z" level=info msg="Node 20d763d5-b44b-40fc-83f3-9976aa6090d6 with Index (197) is Up" 
2017-07-17 20:27:11,081 INFO reaped unknown pid 968
time="2017-07-17T20:27:19Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 95a12bef-eb02-4fd1-b2d9-b457b5f38350). Trying attachSet..." 
time="2017-07-17T20:27:20Z" level=info msg="Failed to attach volume set  vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:23Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2f0b7dfc-30f9-4d2a-a3cc-3155377b78b3). Trying attachSet..." 
time="2017-07-17T20:27:23Z" level=info msg="Failed to attach volume set  vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 9c1b5212-1047-4015-a9b4-717691e7f4a7). Trying attachSet..." 
time="2017-07-17T20:27:26Z" level=info msg="Failed to attach volume set  vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:29Z" level=info msg="Deleted volume vol-011bc83c090da3408" 
time="2017-07-17T20:27:29Z" level=info msg="Deleted volume vol-007d1c99f1baf09ab" 
time="2017-07-17T20:27:30Z" level=info msg="Deleted volume vol-0ba1e6a53ba00a031" 
time="2017-07-17T20:27:30Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1045
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:27:30,813 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:31,816 INFO spawned: 'pxdaemon' with pid 1110
PXPROCS: Started px-storage with pid 1111
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:27:33,430 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1120
PXPROCS: Started watchdog with pid 1121
2017-07-17_20:27:34: PX-Watchdog: Starting watcher
2017-07-17_20:27:34: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid979-time1500323220
2017-07-17_20:27:34: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:00.237474507 +0000 UTC.stack
2017-07-17_20:27:35: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:27:35Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:27:35Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1120  1110 13 20:27 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:27:35: PX-Watchdog: (pid 1120): Begin monitoring
time="2017-07-17T20:27:35Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a8c4a83c-cc21-4111-a26f-488d8ecdecb5). Trying attachSet..." 
time="2017-07-17T20:27:35Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:27:35Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:27:35Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:27:35Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:27:35Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:27:35Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:27:35Z" level=info msg="New NodeID is af5717a7-6264-4758-995b-e72ec7f915cc" 
time="2017-07-17T20:27:35Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:27:35Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:27:35Z" level=info msg="Node af5717a7-6264-4758-995b-e72ec7f915cc with Index (198) is Up" 
2017-07-17 20:27:41,249 INFO reaped unknown pid 1039
time="2017-07-17T20:27:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 172e05e6-8748-4f0b-8011-5d1477c0441a). Trying attachSet..." 
time="2017-07-17T20:27:46Z" level=info msg="Failed to attach volume set  vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: f2f608f7-c284-44df-b4d5-e50a71f9a46a). Trying attachSet..." 
time="2017-07-17T20:27:50Z" level=info msg="Failed to attach volume set  vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: af94b5af-4b5b-4007-a3c4-21cbe37b7f01). Trying attachSet..." 
time="2017-07-17T20:27:53Z" level=info msg="Failed to attach volume set  vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..." 
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0689dea182e8ae258" 
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0d2f6ebed91295e99" 
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0a29d43b2e71318d8" 
time="2017-07-17T20:27:56Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1111
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:27:57,701 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:58,703 INFO spawned: 'pxdaemon' with pid 1176
PXPROCS: Started px-storage with pid 1177
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:00,347 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1186
PXPROCS: Started watchdog with pid 1187
2017-07-17_20:28:01: PX-Watchdog: Starting watcher
2017-07-17_20:28:01: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1045-time1500323250
2017-07-17_20:28:01: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:30.154572756 +0000 UTC.stack
2017-07-17_20:28:01: PX-Watchdog: Waiting for px process to start
root      1186  1176  0 20:28 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:01: PX-Watchdog: (pid 1186): Begin monitoring
time="2017-07-17T20:28:01Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:28:01Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:28:02Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: f0c15705-922f-494a-89cd-84a4a543122d). Trying attachSet..." 
time="2017-07-17T20:28:02Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:28:02Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:28:02Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:28:02Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:28:02Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:28:02Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:28:02Z" level=info msg="New NodeID is 45bb1f10-c864-4d1e-85d8-2164533074ed" 
time="2017-07-17T20:28:02Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:28:02Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:28:02Z" level=info msg="Node 45bb1f10-c864-4d1e-85d8-2164533074ed with Index (199) is Up" 
2017-07-17 20:28:08,335 INFO reaped unknown pid 1104
Waiting for state transition to "attached".
time="2017-07-17T20:28:16Z" level=warning msg="Rollback attach volume /dev/xvdi, Error InvalidVolume.ZoneMismatch: The volume 'vol-037a2b5f2a17e29a3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a0836ff2-f7a1-42f5-b671-56ce701090b9" 
time="2017-07-17T20:28:16Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-037a2b5f2a17e29a3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a0836ff2-f7a1-42f5-b671-56ce701090b9). Trying attachSet..." 
time="2017-07-17T20:28:16Z" level=info msg="Failed to attach volume set  vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..." 
time="2017-07-17T20:28:20Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01d3841fc605f322f is already attached to an instance\n\tstatus code: 400, request id: b07d643c-5963-4675-94c3-55a77a0ce75c). Trying attachSet..." 
time="2017-07-17T20:28:20Z" level=info msg="Failed to attach volume set  vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..." 
time="2017-07-17T20:28:23Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01d3841fc605f322f is already attached to an instance\n\tstatus code: 400, request id: af301d66-945c-44c1-a39c-0e60c957db56). Trying attachSet..." 
time="2017-07-17T20:28:24Z" level=info msg="Failed to attach volume set  vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..." 
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-0bed0efe74e5c82e5" 
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-037a2b5f2a17e29a3" 
time="2017-07-17T20:28:27Z" level=warning msg="Delete failed for volume vol-01d3841fc605f322f, Error VolumeInUse: Volume vol-01d3841fc605f322f is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: f7e4a11b-8f8e-4414-b052-9c2b48e6bce2" 
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-01d3841fc605f322f" 
time="2017-07-17T20:28:27Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1177
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:28:27,492 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:28:28,495 INFO spawned: 'pxdaemon' with pid 1245
PXPROCS: Started px-storage with pid 1246
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:30,130 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1255
PXPROCS: Started watchdog with pid 1256
2017-07-17_20:28:31: PX-Watchdog: Starting watcher
2017-07-17_20:28:31: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1111-time1500323276
2017-07-17_20:28:31: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:56.973406987 +0000 UTC.stack
2017-07-17_20:28:31: PX-Watchdog: Waiting for px process to start
root      1255  1245  0 20:28 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:31: PX-Watchdog: (pid 1255): Begin monitoring
time="2017-07-17T20:28:31Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:28:31Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:28:32Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 63e35a20-78cf-4f60-af8f-9124245cd108). Trying attachSet..." 
time="2017-07-17T20:28:32Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:28:32Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:28:32Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:28:32Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:28:32Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:28:32Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:28:32Z" level=info msg="New NodeID is 77042558-cabb-4d05-877b-d2105e95088b" 
time="2017-07-17T20:28:32Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:28:32Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:28:32Z" level=info msg="Node 77042558-cabb-4d05-877b-d2105e95088b with Index (200) is Up" 
2017-07-17 20:28:38,064 INFO reaped unknown pid 1170
time="2017-07-17T20:28:43Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 307736dd-d7fc-462b-b35c-ff431f41c9e3). Trying attachSet..." 
time="2017-07-17T20:28:43Z" level=info msg="Failed to attach volume set  vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..." 
time="2017-07-17T20:28:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 176322cd-3053-4c15-bd87-e4fe4f930de0). Trying attachSet..." 
time="2017-07-17T20:28:47Z" level=info msg="Failed to attach volume set  vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..." 
time="2017-07-17T20:28:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b20869bc-923c-4a0d-9812-2e56131c407f). Trying attachSet..." 
time="2017-07-17T20:28:50Z" level=info msg="Failed to attach volume set  vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..." 
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-07bd9c31eee1a5369" 
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-04a8c069f6593c478" 
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-07b74ab9dd4c1516e" 
time="2017-07-17T20:28:53Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1246
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:28:53,992 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:28:54,995 INFO spawned: 'pxdaemon' with pid 1311
PXPROCS: Started px-storage with pid 1312
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:56,611 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1321
PXPROCS: Started watchdog with pid 1322
2017-07-17_20:28:58: PX-Watchdog: Starting watcher
2017-07-17_20:28:58: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1177-time1500323307
2017-07-17_20:28:58: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:28:27.364753832 +0000 UTC.stack
2017-07-17_20:28:58: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:28:58Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:28:58Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
root      1321  1311  0 20:28 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:58: PX-Watchdog: (pid 1321): Begin monitoring
time="2017-07-17T20:28:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: ea863fee-bd89-4fbb-b64c-418dd0c8e652). Trying attachSet..." 
time="2017-07-17T20:28:58Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:28:58Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:28:58Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:28:58Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:28:58Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:28:58Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:28:58Z" level=info msg="New NodeID is bcf1228f-b5c9-4bd9-a01b-525cf3e9df9f" 
time="2017-07-17T20:28:58Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:28:58Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:28:58Z" level=info msg="Node bcf1228f-b5c9-4bd9-a01b-525cf3e9df9f with Index (201) is Up" 
2017-07-17 20:29:04,856 INFO reaped unknown pid 1234
Waiting for state transition to "attached".
time="2017-07-17T20:29:12Z" level=warning msg="Rollback attach volume /dev/xvdj, Error InvalidVolume.ZoneMismatch: The volume 'vol-01da48b0445fed392' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 31c3903a-9e85-48ba-9121-74108bdb060b" 
time="2017-07-17T20:29:12Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01da48b0445fed392' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 31c3903a-9e85-48ba-9121-74108bdb060b). Trying attachSet..." 
time="2017-07-17T20:29:12Z" level=info msg="Failed to attach volume set  vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..." 
time="2017-07-17T20:29:15Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00aa97e0ca04b783e is already attached to an instance\n\tstatus code: 400, request id: 5d08c26e-cd8b-4382-a14f-63e176371ea8). Trying attachSet..." 
time="2017-07-17T20:29:16Z" level=info msg="Failed to attach volume set  vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..." 
time="2017-07-17T20:29:19Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00aa97e0ca04b783e is already attached to an instance\n\tstatus code: 400, request id: edb3fe8e-78ce-41f6-9482-b4f677b8a631). Trying attachSet..." 
time="2017-07-17T20:29:19Z" level=info msg="Failed to attach volume set  vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..." 
time="2017-07-17T20:29:22Z" level=info msg="Deleted volume vol-0e600499b93e24abe" 
time="2017-07-17T20:29:23Z" level=info msg="Deleted volume vol-01da48b0445fed392" 
time="2017-07-17T20:29:23Z" level=warning msg="Delete failed for volume vol-00aa97e0ca04b783e, Error VolumeInUse: Volume vol-00aa97e0ca04b783e is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 8b5a2444-114e-4cec-a746-a44d57e0d8bd" 
time="2017-07-17T20:29:23Z" level=info msg="Deleted volume vol-00aa97e0ca04b783e" 
time="2017-07-17T20:29:23Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1312
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:29:23,313 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:29:24,315 INFO spawned: 'pxdaemon' with pid 1376
PXPROCS: Started px-storage with pid 1377
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:29:25,971 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1386
PXPROCS: Started watchdog with pid 1387
2017-07-17_20:29:27: PX-Watchdog: Starting watcher
2017-07-17_20:29:27: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1246-time1500323333
2017-07-17_20:29:27: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:28:53.69988716 +0000 UTC.stack
2017-07-17_20:29:27: PX-Watchdog: Waiting for px process to start
root      1386  1376  0 20:29 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:29:27: PX-Watchdog: (pid 1386): Begin monitoring
time="2017-07-17T20:29:27Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:29:27Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:29:28Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8c76aa23-2d41-47f5-a37f-6184fa1eee55). Trying attachSet..." 
time="2017-07-17T20:29:28Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:29:28Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:29:28Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:29:28Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:29:28Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:29:28Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:29:28Z" level=info msg="New NodeID is d0fc3fea-3470-4ed2-9b9e-bb5d41443015" 
time="2017-07-17T20:29:28Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:29:28Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:29:28Z" level=info msg="Node d0fc3fea-3470-4ed2-9b9e-bb5d41443015 with Index (202) is Up" 
2017-07-17 20:29:34,668 INFO reaped unknown pid 1304
Waiting for state transition to "attached".
time="2017-07-17T20:29:47Z" level=warning msg="Rollback attach volume /dev/xvdk, Error InvalidVolume.ZoneMismatch: The volume 'vol-09ebb76decd2a21a5' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bbfccef8-85a2-41c8-9d91-bc200b9dff9e" 
time="2017-07-17T20:29:47Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-09ebb76decd2a21a5' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bbfccef8-85a2-41c8-9d91-bc200b9dff9e). Trying attachSet..." 
time="2017-07-17T20:29:47Z" level=info msg="Failed to attach volume set  vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..." 
time="2017-07-17T20:29:51Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00a1007e4a44e2349 is already attached to an instance\n\tstatus code: 400, request id: 8f446eb2-bca8-408f-9d7b-90554efe480f). Trying attachSet..." 
time="2017-07-17T20:29:51Z" level=info msg="Failed to attach volume set  vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..." 
time="2017-07-17T20:29:54Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00a1007e4a44e2349 is already attached to an instance\n\tstatus code: 400, request id: ccaf35d4-c687-485f-bac3-725f5690b3d7). Trying attachSet..." 
time="2017-07-17T20:29:55Z" level=info msg="Failed to attach volume set  vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..." 
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-09ebb76decd2a21a5" 
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-0c275e24e94030709" 
time="2017-07-17T20:29:58Z" level=warning msg="Delete failed for volume vol-00a1007e4a44e2349, Error VolumeInUse: Volume vol-00a1007e4a44e2349 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 3d4a0160-ad3e-41c6-adbd-d2d7bfa061b8" 
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-00a1007e4a44e2349" 
time="2017-07-17T20:29:58Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1377
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:29:58,749 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:29:59,752 INFO spawned: 'pxdaemon' with pid 1443
PXPROCS: Started px-storage with pid 1444
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:01,240 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:30:01,241 INFO reaped unknown pid 1369
PXPROCS: Started px with pid 1453
PXPROCS: Started watchdog with pid 1454
2017-07-17_20:30:02: PX-Watchdog: Starting watcher
2017-07-17_20:30:02: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1312-time1500323363
2017-07-17_20:30:02: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:29:23.190405658 +0000 UTC.stack
2017-07-17_20:30:02: PX-Watchdog: Waiting for px process to start
root      1453  1443  0 20:30 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:30:02: PX-Watchdog: (pid 1453): Begin monitoring
time="2017-07-17T20:30:02Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:30:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:30:03Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ba5cfc3-15cb-4f96-90ba-fa2205c7b19c). Trying attachSet..." 
time="2017-07-17T20:30:03Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:30:03Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:30:03Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:30:03Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:30:03Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:30:03Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:30:03Z" level=info msg="New NodeID is a0704554-ad8d-4e45-9233-0e5788efa64d" 
time="2017-07-17T20:30:03Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:30:03Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:30:03Z" level=info msg="Node a0704554-ad8d-4e45-9233-0e5788efa64d with Index (203) is Up" 
time="2017-07-17T20:30:15Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 5a88123a-1ebb-4c14-a4ac-b7af49a13e14). Trying attachSet..." 
time="2017-07-17T20:30:15Z" level=info msg="Failed to attach volume set  vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..." 
time="2017-07-17T20:30:18Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: ca24dbfa-d0ed-4733-b4a4-d7ece3d88138). Trying attachSet..." 
time="2017-07-17T20:30:18Z" level=info msg="Failed to attach volume set  vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..." 
time="2017-07-17T20:30:21Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4cc59be9-e927-4949-a7ce-2d0cbc25bfe5). Trying attachSet..." 
time="2017-07-17T20:30:22Z" level=info msg="Failed to attach volume set  vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..." 
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-002226758ec436a2b" 
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-0c0c0d1b5f73d207c" 
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-0fddd4790e207a76e" 
time="2017-07-17T20:30:25Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1444
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:30:25,593 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:30:26,596 INFO spawned: 'pxdaemon' with pid 1510
PXPROCS: Started px-storage with pid 1511
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:28,238 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1520
PXPROCS: Started watchdog with pid 1521
2017-07-17_20:30:29: PX-Watchdog: Starting watcher
2017-07-17_20:30:29: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1377-time1500323398
2017-07-17_20:30:29: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:29:58.533024037 +0000 UTC.stack
2017-07-17_20:30:29: PX-Watchdog: Waiting for px process to start
root      1520  1510  0 20:30 ?        00:00:00 /usr/local/bin/px -daemon
time="2017-07-17T20:30:29Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:30:29Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-17_20:30:29: PX-Watchdog: (pid 1520): Begin monitoring
2017-07-17 20:30:30,468 INFO reaped unknown pid 1434
time="2017-07-17T20:30:30Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2d90acf5-d8ed-417f-bdcd-02f668f7e8d2). Trying attachSet..." 
time="2017-07-17T20:30:30Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:30:30Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:30:30Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:30:30Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:30:30Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:30:30Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:30:30Z" level=info msg="New NodeID is 1a26f325-6f17-4e8e-bf7e-3891de932909" 
time="2017-07-17T20:30:30Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:30:30Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:30:30Z" level=info msg="Node 1a26f325-6f17-4e8e-bf7e-3891de932909 with Index (204) is Up" 
time="2017-07-17T20:30:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc8301d9-4931-4f69-8b17-97fa2468ace8). Trying attachSet..." 
time="2017-07-17T20:30:45Z" level=info msg="Failed to attach volume set  vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..." 
time="2017-07-17T20:30:48Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 45effe32-fe2f-4d32-bfb2-f5d1c70a055e). Trying attachSet..." 
time="2017-07-17T20:30:48Z" level=info msg="Failed to attach volume set  vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..." 
time="2017-07-17T20:30:51Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0fa50a78-b1e6-4694-a420-1248e013a0a0). Trying attachSet..." 
time="2017-07-17T20:30:51Z" level=info msg="Failed to attach volume set  vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..." 
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-03437c1d61237e06d" 
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-070c06c66a4edf5e3" 
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-0def01de3c342c0b2" 
time="2017-07-17T20:30:54Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1511
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:30:55,281 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:30:56,283 INFO spawned: 'pxdaemon' with pid 1575
PXPROCS: Started px-storage with pid 1576
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:57,914 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1585
PXPROCS: Started watchdog with pid 1586
2017-07-17_20:30:59: PX-Watchdog: Starting watcher
2017-07-17_20:30:59: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1444-time1500323425
2017-07-17_20:30:59: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:30:25.457395001 +0000 UTC.stack
2017-07-17_20:30:59: PX-Watchdog: Waiting for px process to start
root      1585  1575  0 20:30 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:30:59: PX-Watchdog: (pid 1585): Begin monitoring
time="2017-07-17T20:30:59Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:30:59Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:31:00Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0690e408-cb5e-4c52-873d-542b796400eb). Trying attachSet..." 
time="2017-07-17T20:31:00Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:31:00Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:31:00Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:31:00Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:31:00Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:31:00Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:31:00Z" level=info msg="New NodeID is a62882c3-5db6-47dd-aa20-e5036752dd4d" 
time="2017-07-17T20:31:00Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:31:00Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:31:00Z" level=info msg="Node a62882c3-5db6-47dd-aa20-e5036752dd4d with Index (205) is Up" 
2017-07-17 20:31:05,929 INFO reaped unknown pid 1501
time="2017-07-17T20:31:10Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b40dbeb9-7e09-4119-8d50-5335145f5edf). Trying attachSet..." 
time="2017-07-17T20:31:11Z" level=info msg="Failed to attach volume set  vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..." 
time="2017-07-17T20:31:14Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 172f4a11-653c-4ce1-b2b7-79269b9fd632). Trying attachSet..." 
time="2017-07-17T20:31:14Z" level=info msg="Failed to attach volume set  vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..." 
time="2017-07-17T20:31:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c5e717fb-bb32-41bb-be69-326462c6e13c). Trying attachSet..." 
time="2017-07-17T20:31:18Z" level=info msg="Failed to attach volume set  vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..." 
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0aafb49c766743134" 
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0ff134bfb92c759c0" 
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0b1d81a6aeab63d89" 
time="2017-07-17T20:31:21Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1576
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:31:21,813 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:31:22,815 INFO spawned: 'pxdaemon' with pid 1640
PXPROCS: Started px-storage with pid 1641
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:31:24,450 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1650
PXPROCS: Started watchdog with pid 1651
2017-07-17_20:31:25: PX-Watchdog: Starting watcher
2017-07-17_20:31:25: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1511-time1500323454
2017-07-17_20:31:26: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:30:54.91891449 +0000 UTC.stack
time="2017-07-17T20:31:26Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:31:26Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
2017-07-17_20:31:26: PX-Watchdog: Waiting for px process to start
root      1650  1640 15 20:31 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:31:26: PX-Watchdog: (pid 1650): Begin monitoring
time="2017-07-17T20:31:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 5259b7a5-046e-4165-82be-7f1424061637). Trying attachSet..." 
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 2a0b4a24-27ed-4db9-887b-4b0f8695c57d - want 3 vols got 1" 
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:31:26Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:31:26Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:31:26Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:31:26Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:31:26Z" level=info msg="New NodeID is 5e7942f5-cc34-41b3-a383-2002e46dc4c3" 
time="2017-07-17T20:31:26Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:31:26Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:31:26Z" level=info msg="Node 5e7942f5-cc34-41b3-a383-2002e46dc4c3 with Index (206) is Up" 
2017-07-17 20:31:32,844 INFO reaped unknown pid 1569
time="2017-07-17T20:31:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: e82c913d-2f17-429c-87dd-ea7471f7fa3f). Trying attachSet..." 
time="2017-07-17T20:31:38Z" level=info msg="Failed to attach volume set  vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..." 
time="2017-07-17T20:31:41Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2525bc39-3e6c-4bd1-8da6-8990ed2e197d). Trying attachSet..." 
time="2017-07-17T20:31:41Z" level=info msg="Failed to attach volume set  vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..." 
time="2017-07-17T20:31:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a913b333-7d5e-47bd-b034-4649afc28576). Trying attachSet..." 
time="2017-07-17T20:31:44Z" level=info msg="Failed to attach volume set  vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..." 
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-0f4102b45575f73e8" 
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-048e6ebce8493ec84" 
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-0656f33ba1a241e3d" 
time="2017-07-17T20:31:48Z" level=warning msg="Unable to save new node configuration: No sets available" 
PXPROCS: px daemon exited with code: 6 

1641
PXPROCS: dumping core and killing px-storage... 

2017-07-17 20:31:48,861 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:31:49,863 INFO spawned: 'pxdaemon' with pid 1706
PXPROCS: Started px-storage with pid 1707
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping 

C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:31:51,498 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1716
PXPROCS: Started watchdog with pid 1717
2017-07-17_20:31:52: PX-Watchdog: Starting watcher
2017-07-17_20:31:52: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1576-time1500323481
2017-07-17_20:31:53: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:31:21.498835997 +0000 UTC.stack
2017-07-17_20:31:53: PX-Watchdog: Waiting for px process to start
root      1716  1706 10 20:31 ?        00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:31:53: PX-Watchdog: (pid 1716): Begin monitoring
time="2017-07-17T20:31:53Z" level=info msg="Registering [kernel] as a volume driver" 
time="2017-07-17T20:31:53Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b" 
time="2017-07-17T20:31:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fce46d5f-3bf4-4681-a1cb-84576b1d5916). Trying attachSet..." 
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1" 
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1" 
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 2a0b4a24-27ed-4db9-887b-4b0f8695c57d - want 3 vols got 1" 
time="2017-07-17T20:31:53Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set" 
time="2017-07-17T20:31:53Z" level=info msg="Node is not yet initialized" 
time="2017-07-17T20:31:54Z" level=error msg="Unsupported scheduler none given to storage driver" 
time="2017-07-17T20:31:54Z" level=info msg="Generating a new Node ID and index." 
time="2017-07-17T20:31:54Z" level=info msg="New NodeID is a2832162-c916-452b-a9ec-056af976a982" 
time="2017-07-17T20:31:54Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]." 
time="2017-07-17T20:31:54Z" level=info msg="Alerts initialized successfully for this cluster" 
time="2017-07-17T20:31:54Z" level=info msg="Node a2832162-c916-452b-a9ec-056af976a982 with Index (207) is Up" 
2017-07-17 20:32:02,471 INFO reaped unknown pid 1634
time="2017-07-17T20:32:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: dc7130f6-9f18-41d7-bbfe-c869ef99da05). Trying attachSet..." 
time="2017-07-17T20:32:08Z" level=info msg="Failed to attach volume set  vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..." 
time="2017-07-17T20:32:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6708a9f6-2fc0-4cc4-9311-7b988419744a). Trying attachSet..." 
time="2017-07-17T20:32:11Z" level=info msg="Failed to attach volume set  vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..." 
time="2017-07-17T20:32:14Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 86997d13-17d6-4fec-9c23-b6dea2d4248d). Trying attachSet..." 
time="2017-07-17T20:32:14Z" level=info msg="Failed to attach volume set  vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..." 

Bound to wrong ip address

I am trying to run px-dev on a CoreOS system that has DC/OS V1.7.2 installed. DC/OS V1.7.2 introduces a DNS forwarder called 'Spartan'; the way it works is to create a new virtual network interface called 'Spartan' which has it's own IP address; all the entries in /etc/resolv.conf are then pointed at that IP address range. (See below for what ifconfig and /etc/resolv.conf look like in this environment.)

This confuses portworx; when it starts up it seems to think that it should use this interface for communication. This leads to errors:

2016/07/18 21:24:14 [WARN] memberlist 198.51.100.1: Refuting a suspect message (from: df125a7a-e77b-4e5c-a29f-f16f9cba4624)
2016/07/18 21:24:16 [WARN] memberlist: Was able to reach e57572a7-d55f-48eb-89f3-f18ad0d27433 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP
2016/07/18 21:24:21 [WARN] memberlist: Was able to reach fa21d6ef-48dc-43b7-84ff-67eaebb00c05 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP
2016/07/18 21:24:26 [WARN] memberlist: Was able to reach e57572a7-d55f-48eb-89f3-f18ad0d27433 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP

You can also see below that it is using the ip address in the pxctl output.

I had this working before the DC/OS upgrade added this new interface, but now it's stuck in a weird state. Is there any way to tell the container which interface it should bind to?

Thanks,

Craig

sudo /opt/pwx/bin/pxctl status
Status: PX is operational
Node ID: df125a7a-e77b-4e5c-a29f-f16f9cba4624
IP: 198.51.100.1
Local Storage Pool: 14 devices
Device Path Caching Tier Size Used Last-Scan
1 /dev/sdo 5.5 TiB 751 MiB 18 Jul 16 21:15 UTC
10 /dev/sdf 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
11 /dev/sdj 5.5 TiB 1.7 GiB 18 Jul 16 21:15 UTC
12 /dev/sdl 5.5 TiB 1.7 GiB 18 Jul 16 21:15 UTC
13 /dev/sdb 5.5 TiB 739 MiB 18 Jul 16 21:15 UTC
14 /dev/sdg 5.5 TiB 739 MiB 18 Jul 16 21:15 UTC
2 /dev/sda 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
3 /dev/sdh 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
4 /dev/sde 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
5 /dev/sdi 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
6 /dev/sdk 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
7 /dev/sdc 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
8 /dev/sdd 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
9 /dev/sdm 5.5 TiB 731 MiB 18 Jul 16 21:15 UTC
total - 76 TiB 12 GiB
Cluster Summary
ID: 5ac2ed6f-7e4e-4e1d-8e8c-3a6df1fb7801
IP: 198.51.100.1 - Capacity: 77 TiB/19 MiB OK (This node)
IP: 10.1.20.1 - Capacity: 77 TiB/19 MiB OK
IP: 10.1.19.1 - Capacity: 77 TiB/173 MiB OK
IP: 10.1.18.1 - Capacity: 77 TiB/19 MiB OK
Global Storage Pool
Total Capacity : 308 TiB
Total Used : 230 MiB

Craig

montana@node-1-17 ~ $ ifconfig spartan
spartan: flags=195<UP,BROADCAST,RUNNING,NOARP> mtu 1500
inet 198.51.100.1 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::48e4:78ff:fef6:bfb prefixlen 64 scopeid 0x20
ether 4a:e4:78:f6:0b:fb txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21972 bytes 7314572 (6.9 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

montana@node-1-17 ~ $ cat /etc/resolv.conf

Generated by gen_resolvconf.py. Do not edit.

Change configuration options by changing DCOS cluster configuration.

This file must be overwritten regularly for proper cluster operation around

master failure.

options timeout:1
options attempts:3

nameserver 198.51.100.1
nameserver 198.51.100.2
nameserver 198.51.100.3

Can't initialize pwx-dev with ETCD SSL

{ "alertingurl": "", "cafile": "/etc/pwx/server.ca.crt", "certfile": "/etc/pwx/client.crt", "certkey": "/etc/pwx/client.key.insecure", "clusterid": "mycluster", "dataiface": "flannel.1", "kvdb": [ "etcd://my.etcd:2379" ], "mgtiface": "flannel.1", "scheduler": "none", "storage": { "devices": [ "/dev/mylvm/portworx" ] }, "version": "1.0" }
it seems portworx don't detect etcd is over SSL, I get this error:
Error in obtaining etcd version: Get http://my.etcd:2379/version: malformed HTTP response \"\\x15\\x03\\x01\\x00\\x02\\x02\\

License?

Is this open source?

pvc pending

Hi
I am testing portworx on GKE
I following your example of cassandra (https://docs.portworx.com/scheduler/kubernetes/cassandra-k8s.html)

The first pod(cassandra-0) is assigned a volume well, but second pod(cassandra-1) is not
pvc is still pending
What should I check more ?

infordb_park@speedy-area-195923:~$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
cassandra-data-cassandra-0 Bound pvc-b0d2eb75-1f68-11e8-b061-42010a9200c5 1Gi RWO px-storageclass 22h
cassandra-data-cassandra-1 Pending px-storageclass 22h

infordb_park@speedy-area-195923:~$ kubectl describe pvc cassandra-data-cassandra-1
Name: cassandra-data-cassandra-1
Namespace: default
StorageClass: px-storageclass
Status: Pending
Volume:
Labels: app=cassandra
Annotations: volume.beta.kubernetes.io/storage-class=px-storageclass
volume.beta.kubernetes.io/storage-provisioner=kubernetes.io/portworx-volume
Capacity:
Access Modes:
Events:
Type Reason Age From Message

---

Warning ProvisioningFailed 2m (x5382 over 22h) persistentvolume-controller Failed to provision volume with StorageClass "px-storageclass": Could not find enough online nodes with sufficient storage at requested Cos (HIGH) level. 2 node(s)
were excluded to enforce group provisioning.
Warning ProvisioningFailed 2m (x5382 over 22h) persistentvolume-controller Failed to provision volume with StorageClass "px-storageclass": Get http://10.55.252.27:9001/v1/osd-volumes/versions: dial tcp 10.55.252.27:9001: i/o timeout
infordb_park@speedy-area-195923:~$

root@gke-cluster-1-default-pool-ea79223e-6srv:~# /opt/pwx/bin/pxctl status
Status: PX is operational
License: Trial (expires in 30 days)
Node ID: gke-cluster-1-default-pool-ea79223e-6srv
IP: 10.146.0.5
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 LOW raid0 10 GiB 2.1 GiB Online default default
Local Storage Devices: 1 device
Device Path Media Type Size Last-Scan
0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 10 GiB 04 Mar 18 04:30 UTC
total - 10 GiB
Cluster Summary
Cluster ID: mycluster
Cluster UUID: df97b45c-a095-4568-bd55-a120badaf66a
Nodes: 3 node(s) with storage (3 online)
IP ID StorageNode Used Capacity Status
10.146.0.3 gke-cluster-1-default-pool-ea79223e-n81h Yes 2.1 GiB 10 GiB Online
10.146.0.2 gke-cluster-1-default-pool-ea79223e-gqkr Yes 2.0 GiB 10 GiB Online
10.146.0.5 gke-cluster-1-default-pool-ea79223e-6srv Yes 2.1 GiB 10 GiB Online (This node)
Global Storage Pool
Total Used : 6.2 GiB
Total Capacity : 30 GiB

CoreOS + Swarm - Volume Plugin not added autm.

I followed the Instruction to install Portworx on Coreos.
https://docs.portworx.com/install/coreos.html

docker info
does not shows the pxd Driver!

After doing something like
docker run -it -v my_pxd_vol:/test busybox echo foo
docker info now shows the pxd volume and i can start using it with swarm.

That occours after every reboot.. very anoying!

Support for Consul

Would love to see support for Consul planned.

Strip binaries

Please strip the PX binaries in portworx/px-dev. The image size may reduce by ~80MiB.

core@master-3 ~ $ docker run --rm --net=none --entrypoint=bash portworx/px-dev -c 'cd /usr/local/bin && bins=$(file * | grep "not stripped" | cut -f1 -d:); du -csh $bins; for bin in $bins; do file $bin; ls -lh $bin; strip $bin; ls -lh $bin; done; du -csh $bins'
81M	px
13M	px-ns
27M	px-obj
28M	px-storage
484K	pxd
2.5M	pxd-dummy
6.3M	pxexec
2.9M	tp
159M	total
px: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=106cff5a7ff79ac56a96a20513d8b7d878bfe465, not stripped
-rwxr-xr-x. 1 root root 81M May 18 22:02 px
-rwxr-xr-x. 1 root root 44M May 23 09:20 px
px-ns: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ed8afabf042430c954f47426a6141fb6aa48f627, not stripped
-rwxr-xr-x. 1 root root 13M May 18 22:02 px-ns
-rwxr-xr-x. 1 root root 7.5M May 23 09:20 px-ns
px-obj: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, not stripped
-rwxr-xr-x. 1 root root 27M May 18 22:02 px-obj
-rwxr-xr-x. 1 root root 17M May 23 09:20 px-obj
px-storage: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=f7459dcc81ca6fdf3a1c0407d9444a53c4ab74b6, not stripped
-rwxr-xr-x. 1 root root 28M May 18 22:02 px-storage
-rwxr-xr-x. 1 root root 2.9M May 23 09:20 px-storage
pxd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=d72fc35b6cb656cc6757e60fd620876e809a5df0, not stripped
-rwxr-xr-x. 1 root root 477K May 18 22:02 pxd
-rwxr-xr-x. 1 root root 27K May 23 09:20 pxd
pxd-dummy: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, not stripped
-rwxr-xr-x. 1 root root 2.5M May 18 22:02 pxd-dummy
-rwxr-xr-x. 1 root root 1.6M May 23 09:20 pxd-dummy
pxexec: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, not stripped
-rwxr-xr-x. 1 root root 6.3M May 18 22:02 pxexec
-rwxr-xr-x. 1 root root 4.0M May 23 09:20 pxexec
tp: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=230a6606707bf419543476ed361c2e47a92b30a5, not stripped
-rwxr-xr-x. 1 root root 2.9M May 18 22:02 tp
-rwxr-xr-x. 1 root root 200K May 23 09:20 tp
44M	px
7.5M	px-ns
17M	px-obj
2.9M	px-storage
32K	pxd
1.6M	pxd-dummy
4.0M	pxexec
204K	tp
77M	total

Error launch px-dev in Rancher 1.2 + RancherOS 0.7.0

After launch the portworx service from Rancher Catalog, the services doesn't start.

Here are the logs from one container. I've enabled kernel-headers in rancheros before start the stack.

12/7/2016 1:48:57 PMWed Dec  7 15:48:57 UTC 2016 : Running on Linux ip-10-0-3-75.sa-east-1.compute.internal 4.4.21-rancher #1 SMP Sat Oct 15 07:53:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
12/7/2016 1:48:57 PMRequired size for /dev/shm is  293601280 is less than 67108864
12/7/2016 1:48:57 PMchecking /usr/src/kernels/4.4.21-rancher
12/7/2016 1:48:57 PMchecking /usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:48:57 PMchecking /lib/modules/4.4.21-rancher/build
12/7/2016 1:48:57 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:48:57 PMchecking /usr/src/linux
12/7/2016 1:48:57 PMLoaded plugins: fastestmirror
12/7/2016 1:48:57 PMRepodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
12/7/2016 1:49:01 PMhttp://mirror.globo.com/epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
12/7/2016 1:49:01 PMTrying other mirror.
12/7/2016 1:49:03 PMhttps://mirror.cedia.org.ec/fedora-epel/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml does not match metalink for epel
12/7/2016 1:49:03 PMTrying other mirror.
12/7/2016 1:49:10 PMDetermining fastest mirrors
12/7/2016 1:49:10 PM * base: centos.xpg.com.br
12/7/2016 1:49:10 PM * epel: mirror.globo.com
12/7/2016 1:49:10 PM * extras: centos.xpg.com.br
12/7/2016 1:49:10 PM * updates: centos.xpg.com.br
12/7/2016 1:49:13 PMNo package kernel-headers-4.4.21-rancher available.
12/7/2016 1:49:13 PMError: Nothing to do
12/7/2016 1:49:13 PMLoaded plugins: fastestmirror
12/7/2016 1:49:13 PMLoading mirror speeds from cached hostfile
12/7/2016 1:49:13 PM * base: centos.xpg.com.br
12/7/2016 1:49:13 PM * epel: mirror.globo.com
12/7/2016 1:49:13 PM * extras: centos.xpg.com.br
12/7/2016 1:49:13 PM * updates: centos.xpg.com.br
12/7/2016 1:49:13 PMNo package kernel-devel-4.4.21-rancher available.
12/7/2016 1:49:13 PMError: Nothing to do
12/7/2016 1:49:13 PMchecking /usr/src/kernels/4.4.21-rancher
12/7/2016 1:49:13 PMchecking /usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:13 PMchecking /lib/modules/4.4.21-rancher/build
12/7/2016 1:49:13 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:13 PMchecking /usr/src/linux
12/7/2016 1:49:13 PMChecking short path for KVERSION 4.4.21-rancher.el7.x86_64
12/7/2016 1:49:13 PMchecking /usr/src/kernels/4.4.21-rancher.el7.x86_64
12/7/2016 1:49:13 PMchecking /usr/src/linux-headers-4.4.21-rancher.el7.x86_64
12/7/2016 1:49:13 PMchecking /lib/modules/4.4.21-rancher.el7.x86_64/build
12/7/2016 1:49:13 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher.el7.x86_64
12/7/2016 1:49:13 PMchecking /usr/src/linux
12/7/2016 1:49:13 PMLoaded plugins: fastestmirror
12/7/2016 1:49:13 PMLoading mirror speeds from cached hostfile
12/7/2016 1:49:13 PM * base: centos.xpg.com.br
12/7/2016 1:49:13 PM * epel: mirror.globo.com
12/7/2016 1:49:13 PM * extras: centos.xpg.com.br
12/7/2016 1:49:13 PM * updates: centos.xpg.com.br
12/7/2016 1:49:14 PMNo package kernel-headers-4.4.21-rancher.el7.x86_64 available.
12/7/2016 1:49:14 PMError: Nothing to do
12/7/2016 1:49:14 PMLoaded plugins: fastestmirror
12/7/2016 1:49:14 PMLoading mirror speeds from cached hostfile
12/7/2016 1:49:14 PM * base: centos.xpg.com.br
12/7/2016 1:49:14 PM * epel: mirror.globo.com
12/7/2016 1:49:14 PM * extras: centos.xpg.com.br
12/7/2016 1:49:14 PM * updates: centos.xpg.com.br
12/7/2016 1:49:14 PMNo package kernel-devel-4.4.21-rancher.el7.x86_64 available.
12/7/2016 1:49:14 PMError: Nothing to do
12/7/2016 1:49:14 PMchecking /usr/src/kernels/4.4.21-rancher.el7.x86_64
12/7/2016 1:49:14 PMchecking /usr/src/linux-headers-4.4.21-rancher.el7.x86_64
12/7/2016 1:49:14 PMchecking /lib/modules/4.4.21-rancher.el7.x86_64/build
12/7/2016 1:49:14 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher.el7.x86_64
12/7/2016 1:49:14 PMchecking /usr/src/linux
12/7/2016 1:49:14 PMchecking /usr/src/kernels/4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /lib/modules/4.4.21-rancher/build
12/7/2016 1:49:14 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux
12/7/2016 1:49:14 PMchecking /usr/src/kernels/4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /lib/modules/4.4.21-rancher/build
12/7/2016 1:49:14 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux
12/7/2016 1:49:14 PMchecking /usr/src/kernels/4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /lib/modules/4.4.21-rancher/build
12/7/2016 1:49:14 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux
12/7/2016 1:49:14 PMchecking /usr/src/kernels/4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /lib/modules/4.4.21-rancher/build
12/7/2016 1:49:14 PMchecking /tmp/kernel_headers/usr/src/linux-headers-4.4.21-rancher
12/7/2016 1:49:14 PMchecking /usr/src/linux
12/7/2016 1:49:14 PMFailed to install/locate kernel headers for 4.4.21-rancher
12/7/2016 1:49:14 PMFailed to load PX module dependencies

Deployment steps for k8s (and probably others)

Hi guys,

It's unclear how to specify to px-dev which block devices it should use to store data/volumes on per node basis.

"socket not connected" inside docker container after recover

Setup:

• Rancher
• 5 Node etcd Cluster from catalog. Heartbeat 5ms, Timeout 75ms
• etcd-proxy on each host "load balancing" for the etcd cluster
• Portworx on 3 Nodes from catalog

Problem:
Despite my best efforts the etcd cluster from the rancher catalog still becomes unavailable for a few milliseconds from time to time.
This causes portworx to restart and recover, which works fine and reports all 3 Nodes as healthy after a few moments.
Execing into a container using a pxd Volume and trying to access said volume still shows the error:
ls: /path/to/file Socket not connected
This is only fixed after restarting the container.

Expected:
It would be best if the storage would simply be accessible again after portworx has recovered. If that is not possible triggering containers which were affected to restart would at least cause the error to self-heal all the way through

How to Rebuild pwx.ko on airgapped server?

I'm trying to run the container on CoreOS 1010.5.0 on an airgapped server (our clusters have an internal registry - nodes do not have access to the 'internet'.

It looks like the container rebuilds the pwx.ko as part of the startup. This is failing for me because I am running the container on an airgapped server that does not have a path back to the various mirrors.

What's the solution?

Some snippets from the log:

...
Cannot find a valid baseurl for repo: base/7/x86_64
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
....

time="2016-06-30T22:14:01Z" level=error msg="Kernel Driver error detected... please load the correct PX kernel module and restart: Kernel Driver Error"
2016-06-30 22:14:01,264 INFO exited: px (exit status 255; not expected)

px-dev hangs hosts, prevents restarts

Setup:
3 Hosts running Portworx.
docker version: 17.03.1-ce, 17.03.2-ce
Portworx Version: portworx/px-dev:1.2.8
Each has 1 Volume at on a volumegroup, which is raid1 on 2 ssd disks
A volume directly on a 3TB HDD was supposed to be added.

Process:
Following https://docs.portworx.com/maintain/scale-up.html:
sudo /opt/pwx/bin/pxctl service maintenance --enter
-- Wait until portworx is back up in maintenance mode --
sudo /opt/pwx/bin/pxctl service drive add /dev/sdc
sudo /opt/pwx/bin/pxctl service maintenance --exit
-- Wait for portworx to return to normal mode --

Portworx stayed in initializing for about 3 minutes, until all 3 Hosts died simultanously with the following log:
Host which was supposed to receive more storage:

[268973.752296] pxd_control_open: pxd-control-0 open OK
[268973.752331] pxd_process_init_reply: pxd-control-0:5048845 init OK
[268973.775532] blk_update_request: I/O error, dev pxd/pxd226604244376265636, sector 0
[268973.776890] blk_update_request: I/O error, dev pxd/pxd226604244376265636, sector 0
[268973.776951] blk_update_request: I/O error, dev pxd/pxd226604244376265636, sector 21241864
[268973.776996] Buffer I/O error on dev pxd/pxd226604244376265636, logical block 2655233, lost sync page write
[268973.777048] JBD2: Error -5 detected when updating journal superblock for pxd!pxd226604244376265636-8.
[268973.777096] Aborting journal on device pxd!pxd226604244376265636-8.
[268973.777134] blk_update_request: I/O error, dev pxd/pxd226604244376265636, sector 21241864
[268973.777179] Buffer I/O error on dev pxd/pxd226604244376265636, logical block 2655233, lost sync page write
[268973.777229] JBD2: Error -5 detected when updating journal superblock for pxd!pxd226604244376265636-8.
[268973.777301] blk_update_request: I/O error, dev pxd/pxd226604244376265636, sector 0
[268973.777345] Buffer I/O error on dev pxd/pxd226604244376265636, logical block 0, lost sync page write
[268973.777395] EXT4-fs error (device pxd!pxd226604244376265636): ext4_put_super:842: Couldn't clean up the journal
[268973.777444] EXT4-fs (pxd!pxd226604244376265636): Remounting filesystem read-only
[268973.777488] EXT4-fs (pxd!pxd226604244376265636): previous I/O error to superblock detected
[268973.777542] blk_update_request: I/O error, dev pxd/pxd226604244376265636, sector 0
[268973.777586] Buffer I/O error on dev pxd/pxd226604244376265636, logical block 0, lost sync page write
[269032.308667] docker0: port 33(vethr2da89069fb) entered disabled state
[269032.324781] device vethr2da89069fb left promiscuous mode
[269032.324811] docker0: port 33(vethr2da89069fb) entered disabled state
[269297.629050] INFO: task jbd2/pxd!pxd640:19027 blocked for more than 300 seconds.
[269297.629098]       Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1
[269297.629124] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[269297.629168] jbd2/pxd!pxd640 D    0 19027      2 0x00000000
[269297.629197]  ffff8866a1f77800 0000000000000000 ffff8874384b50c0 ffff886f885b5140
[269297.629247]  ffff88743f218700 ffff94830a79fb20 ffffffffa61f784d ffffffffa5ef957c
[269297.629296]  ffff886cb7ee9c00 0000000000000034 0000000055e34000 ffff886f885b5140
[269297.629345] Call Trace:
[269297.629371]  [<ffffffffa61f784d>] ? __schedule+0x23d/0x6d0
[269297.629399]  [<ffffffffa5ef957c>] ? blk_rq_init+0xbc/0xd0
[269297.629425]  [<ffffffffa61f8580>] ? bit_wait_timeout+0x90/0x90
[269297.629451]  [<ffffffffa61f7d12>] ? schedule+0x32/0x80
[269297.629477]  [<ffffffffa61fb249>] ? schedule_timeout+0x249/0x300
[269297.629505]  [<ffffffffa5efe01f>] ? blk_peek_request+0x5f/0x290
[269297.629532]  [<ffffffffa61f8580>] ? bit_wait_timeout+0x90/0x90
[269297.629559]  [<ffffffffa61f7594>] ? io_schedule_timeout+0xb4/0x130
[269297.629587]  [<ffffffffa5cbb4f7>] ? prepare_to_wait+0x57/0x80
[269297.629613]  [<ffffffffa61f8597>] ? bit_wait_io+0x17/0x60
[269297.629639]  [<ffffffffa61f808c>] ? __wait_on_bit+0x5c/0x90
[269297.629665]  [<ffffffffa61f8580>] ? bit_wait_timeout+0x90/0x90
[269297.629692]  [<ffffffffa61f81ee>] ? out_of_line_wait_on_bit+0x7e/0xa0
[269297.629720]  [<ffffffffa5cbb820>] ? autoremove_wake_function+0x40/0x40
[269297.629753]  [<ffffffffc04e5d18>] ? jbd2_journal_commit_transaction+0xd48/0x17e0 [jbd2]
[269297.629798]  [<ffffffffa5cafd67>] ? put_prev_entity+0x47/0x840
[269297.629826]  [<ffffffffa5c2476b>] ? __switch_to+0x2bb/0x700
[269297.629855]  [<ffffffffa5ce65cd>] ? try_to_del_timer_sync+0x4d/0x80
[269297.629884]  [<ffffffffc04ea9ed>] ? kjournald2+0xdd/0x280 [jbd2]
[269297.629911]  [<ffffffffa5cbb7e0>] ? wake_up_atomic_t+0x30/0x30
[269297.629939]  [<ffffffffc04ea910>] ? commit_timeout+0x10/0x10 [jbd2]
[269297.629968]  [<ffffffffa5c97520>] ? kthread+0xe0/0x100
[269297.629993]  [<ffffffffa5c2476b>] ? __switch_to+0x2bb/0x700
[269297.630019]  [<ffffffffa5c97440>] ? kthread_park+0x60/0x60
[269297.630046]  [<ffffffffa61fc835>] ? ret_from_fork+0x25/0x30
[269297.630072] NMI backtrace for cpu 7
[269297.630095] CPU: 7 PID: 81 Comm: khungtaskd Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.18-1~bpo8+1
[269297.630143] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.2.T201502251406 02/25/2015
[269297.630194]  0000000000000000 ffffffffa5f29dd5 0000000000000000 0000000000000007
[269297.630243]  ffffffffa5f2e300 0000000000000007 ffffffffa5c4cae0 ffff886f885b5140
[269297.630292]  ffffffffa5f2e40a ffff886f885b5140 00000000003ff18f ffffffffa5d27a90
[269297.630341] Call Trace:
[269297.630363]  [<ffffffffa5f29dd5>] ? dump_stack+0x5c/0x77
[269297.630389]  [<ffffffffa5f2e300>] ? nmi_cpu_backtrace+0x90/0xa0
[269297.630417]  [<ffffffffa5c4cae0>] ? irq_force_complete_move+0x140/0x140
[269297.630445]  [<ffffffffa5f2e40a>] ? nmi_trigger_cpumask_backtrace+0xfa/0x130
[269297.630474]  [<ffffffffa5d27a90>] ? watchdog+0x2b0/0x330
[269297.630499]  [<ffffffffa5d277e0>] ? reset_hung_task_detector+0x10/0x10
[269297.630528]  [<ffffffffa5c97520>] ? kthread+0xe0/0x100
[269297.630553]  [<ffffffffa5c2476b>] ? __switch_to+0x2bb/0x700
[269297.630579]  [<ffffffffa5c97440>] ? kthread_park+0x60/0x60
[269297.630606]  [<ffffffffa61fc835>] ? ret_from_fork+0x25/0x30
[269297.630632] Sending NMI from CPU 7 to CPUs 0-6,8-11:
[269297.630666] NMI backtrace for cpu 8 skipped: idling at pc 0xffffffffa61fc02e
[269297.630697] NMI backtrace for cpu 2 skipped: idling at pc 0xffffffffa61fc02e
[269297.630727] NMI backtrace for cpu 10 skipped: idling at pc 0xffffffffa61fc02e
[269297.630772] NMI backtrace for cpu 11 skipped: idling at pc 0xffffffffa61fc02e
[269297.630817] NMI backtrace for cpu 1 skipped: idling at pc 0xffffffffa61fc02e
[269297.630847] NMI backtrace for cpu 4
[269297.630871] CPU: 4 PID: 5280 Comm: confd Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.18-1~bpo8+1
[269297.630919] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.2.T201502251406 02/25/2015
[269297.630971] task: ffff886f9e4c4000 task.stack: ffff948307674000
[269297.630998] RIP: 0033:[<000000000045f98b>] c [<000000000045f98b>] 0x45f98b
[269297.631026] RSP: 002b:000000c82004e9e8  EFLAGS: 00000283
[269297.631052] RAX: 000000c820a2f630 RBX: 000000000045f6a0 RCX: 0000000000000047
[269297.631095] RDX: 000000c82004ea00 RSI: 000000c8218854b8 RDI: 000000c822363518
[269297.631138] RBP: 0000000000bb5d00 R08: 000000c8209069c0 R09: 000000c8209069c0
[269297.631181] R10: 000000c822363518 R11: 0000000000000000 R12: 0000000000000005
[269297.631224] R13: 0000000000a282b8 R14: 0000000000000004 R15: 0000000000000008
[269297.631267] FS:  000000c82003a868(0000) GS:ffff88743f300000(0000) knlGS:0000000000000000
[269297.631311] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[269297.631338] CR2: 000000c4282fa0d0 CR3: 0000000b97b4c000 CR4: 00000000001406e0
[269297.631382] NMI backtrace for cpu 5
[269297.631405] CPU: 5 PID: 13343 Comm: confd Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.18-1~bpo8+1
[269297.634145] Hardware name: Supermicro X9SRE/X9SRE-3F/X9SRi/X9SRi-3F/X9SRE/X9SRE-3F/X9SRi/X9SRi-3F, BIOS 3.2.T201502251406 02/25/2015
[269297.634196] task: ffff886f86685080 task.stack: ffff948324b4c000
[269297.634222] RIP: 0033:[<000000000041338b>] c [<000000000041338b>] 0x41338b
[269297.634250] RSP: 002b:000000c821bd9820  EFLAGS: 00000293
[269297.634276] RAX: 0000000000080000 RBX: 00007fe7706614b0 RCX: 000000c820001200
[269297.634318] RDX: 000000c821ddf2c0 RSI: 0000000000000007 RDI: 000000c821ddf262
[269297.634360] RBP: 0000000000140dfc R08: 0000000000140dfc R09: 000000c821ddf262
[269297.634402] R10: 0000000000000002 R11: 000000c821bd9ce0 R12: 0000000000000032
[269297.634444] R13: 0000000000a28614 R14: 000000000000000a R15: 0000000000000008
[269297.634486] FS:  000000c8205b2068(0000) GS:ffff88743f340000(0000) knlGS:0000000000000000
[269297.634530] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[269297.634556] CR2: 000000c427dd0d00 CR3: 0000000e172a4000 CR4: 00000000001406e0
[269297.634600] NMI backtrace for cpu 6 skipped: idling at pc 0xffffffffa61fc02e
[269297.634630] NMI backtrace for cpu 0 skipped: idling at pc 0xffffffffa61fc02e
[269297.634661] NMI backtrace for cpu 3 skipped: idling at pc 0xffffffffa61fc02e
[269297.634691] NMI backtrace for cpu 9 skipped: idling at pc 0xffffffffa61fc02e
[269297.634739] Kernel panic - not syncing: hung_task: blocked tasks

Other hosts:

[268342.182875] INFO: task px-ns:22180 blocked for more than 300 seconds.
[268342.182917]       Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1
[268342.182954] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[268342.183015] px-ns           D    0 22180  22038 0x00000000
[268342.183054]  ffff89dbd8445800 ffff89ddf3fa5c00 ffff89dc61fe9080 ffff89dea134a000
[268342.184314]  ffff89deee498700 ffff9fbc87d37d40 ffffffffb43f784d ffff89db4fdac080
[268342.184404]  00000000fffffffb 00000000b40fe01f ffff89db588da800 ffff89dea134a000
[268342.184467] Call Trace:
[268342.184500]  [<ffffffffb43f784d>] ? __schedule+0x23d/0x6d0
[268342.184538]  [<ffffffffb4036e80>] ? do_fsync+0x60/0x60
[268342.184573]  [<ffffffffb43f7d12>] ? schedule+0x32/0x80
[268342.184621]  [<ffffffffb43fb249>] ? schedule_timeout+0x249/0x300
[268342.184660]  [<ffffffffb40f95ef>] ? __blk_run_queue+0x2f/0x40
[268342.184698]  [<ffffffffb40fe92a>] ? blk_queue_bio+0x39a/0x3b0
[268342.184746]  [<ffffffffb4036e80>] ? do_fsync+0x60/0x60
[268342.184794]  [<ffffffffb43f7594>] ? io_schedule_timeout+0xb4/0x130
[268342.184831]  [<ffffffffb43f912a>] ? wait_for_completion_io+0xfa/0x130
[268342.184872]  [<ffffffffb3ea2b70>] ? wake_up_q+0x60/0x60
[268342.184908]  [<ffffffffb40f37dc>] ? submit_bio_wait+0x5c/0x80
[268342.184944]  [<ffffffffb4100233>] ? blkdev_issue_flush+0x63/0x90
[268342.184998]  [<ffffffffc066cdfa>] ? ext4_sync_fs+0x14a/0x1c0 [ext4]
[268342.185050]  [<ffffffffb4007007>] ? iterate_supers+0xb7/0x110
[268342.185088]  [<ffffffffb4036f52>] ? sys_sync+0x62/0xb0
[268342.185124]  [<ffffffffb43fc5bb>] ? system_call_fast_compare_end+0xc/0x9b
[268342.185166] NMI backtrace for cpu 4
[268342.185198] CPU: 4 PID: 57 Comm: khungtaskd Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.18-1~bpo8+1
[268342.185268] Hardware name: FUJITSU  /D3401-H2, BIOS V5.0.0.12 R1.5.0 for D3401-H2x                     02/27/2017
[268342.185334]  0000000000000000 ffffffffb4129dd5 0000000000000000 0000000000000004
[268342.185399]  ffffffffb412e300 0000000000000004 ffffffffb3e4cae0 ffff89dc61b8b0c0
[268342.185465]  ffffffffb412e40a ffff89dea134a000 00000000003ffcda ffffffffb3f27a90
[268342.185528] Call Trace:
[268342.185558]  [<ffffffffb4129dd5>] ? dump_stack+0x5c/0x77
[268342.185593]  [<ffffffffb412e300>] ? nmi_cpu_backtrace+0x90/0xa0
[268342.185633]  [<ffffffffb3e4cae0>] ? irq_force_complete_move+0x140/0x140
[268342.185673]  [<ffffffffb412e40a>] ? nmi_trigger_cpumask_backtrace+0xfa/0x130
[268342.185713]  [<ffffffffb3f27a90>] ? watchdog+0x2b0/0x330
[268342.185749]  [<ffffffffb3f277e0>] ? reset_hung_task_detector+0x10/0x10
[268342.185789]  [<ffffffffb3e97520>] ? kthread+0xe0/0x100
[268342.185826]  [<ffffffffb3e2476b>] ? __switch_to+0x2bb/0x700
[268342.185862]  [<ffffffffb3e97440>] ? kthread_park+0x60/0x60
[268342.185902]  [<ffffffffb43fc835>] ? ret_from_fork+0x25/0x30
[268342.185939] Sending NMI from CPU 4 to CPUs 0-3,5-7:
[268342.185988] NMI backtrace for cpu 0 skipped: idling at pc 0xffffffffb43fc02e
[268342.186030] NMI backtrace for cpu 7 skipped: idling at pc 0xffffffffb43fc02e
[268342.186074] NMI backtrace for cpu 3 skipped: idling at pc 0xffffffffb43fc02e
[268342.186116] NMI backtrace for cpu 2 skipped: idling at pc 0xffffffffb43fc02e
[268342.186159] NMI backtrace for cpu 6 skipped: idling at pc 0xffffffffb43fc02e
[268342.186201] NMI backtrace for cpu 1 skipped: idling at pc 0xffffffffb43fc02e
[268342.186244] NMI backtrace for cpu 5
[268342.186277] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G           OE   4.9.0-0.bpo.2-amd64 #1 Debian 4.9.18-1~bpo8+1
[268342.186344] Hardware name: FUJITSU  /D3401-H2, BIOS V5.0.0.12 R1.5.0 for D3401-H2x                     02/27/2017
[268342.186412] task: ffff89dea73eb000 task.stack: ffff9fbc862f0000
[268342.186450] RIP: 0010:[<ffffffffb432ac4c>] c [<ffffffffb432ac4c>] netlink_has_listeners+0xc/0x60
[268342.186512] RSP: 0018:ffff89deee543ca0  EFLAGS: 00000202
[268342.186548] RAX: 0000000000000006 RBX: ffff89d8b29b0040 RCX: 0000000000000001
[268342.186606] RDX: ffff89dabaa42980 RSI: 0000000000000001 RDI: ffff89dab19d0000
[268342.186663] RBP: ffff89d8b29b00c8 R08: 0000000000000002 R09: 0000000000000000
[268342.186722] R10: 0000000000000000 R11: 0000000000000000 R12: ffff89cf9c9a440e
[268342.186781] R13: ffff89cf9c9a43fa R14: ffff89d8b29b0040 R15: ffff89d8b29b0040
[268342.186839] FS:  0000000000000000(0000) GS:ffff89deee540000(0000) knlGS:0000000000000000
[268342.186899] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[268342.186934] CR2: 00007f41cce8b3a0 CR3: 0000000d54e9b000 CR4: 00000000003406e0
[268342.186992] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[268342.187050] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[268342.187108] Stack:
[268342.187134]  ffffffffb42df023c ffff89d464abf100c ffffffffb435ddecc fcf7eefb2401a738c
[268342.187195]  ffff89d31c1a8000c ffff89d464abf100c ffff89d464abf100c ffff89dbfd99a000c
[268342.187255]  ffffffff00000000c ffff89deee543d58c ffff89deee543d28c fcf7eefb2401a738c
[268342.187315] Call Trace:
[268342.187342]  <IRQ> d [<ffffffffb42df023>] ? __sk_free+0x73/0xa0
[268342.187378]  [<ffffffffb435ddec>] ? tcp_v4_rcv+0x8bc/0x9e0
[268342.187413]  [<ffffffffb43378bb>] ? ip_local_deliver_finish+0x8b/0x1c0
[268342.187451]  [<ffffffffb4337b8b>] ? ip_local_deliver+0x6b/0xf0
[268342.187486]  [<ffffffffb435d502>] ? tcp_v4_early_demux+0x112/0x140
[268342.187522]  [<ffffffffb4337830>] ? ip_rcv_finish+0x3e0/0x3e0
[268342.187558]  [<ffffffffb4337e91>] ? ip_rcv+0x281/0x3b0
[268342.187591]  [<ffffffffb4337450>] ? inet_del_offload+0x40/0x40
[268342.187627]  [<ffffffffb42f6cce>] ? __netif_receive_skb_core+0x2be/0xa40
[268342.187665]  [<ffffffffb42f8532>] ? process_backlog+0x92/0x140
[268342.187701]  [<ffffffffb42f7ca5>] ? net_rx_action+0x245/0x380
[268342.187736]  [<ffffffffb43ff0e6>] ? __do_softirq+0x106/0x292
[268342.187771]  [<ffffffffb3e7dbb8>] ? irq_exit+0x98/0xa0
[268342.187805]  [<ffffffffb43fee2f>] ? do_IRQ+0x4f/0xd0
[268342.187839]  [<ffffffffb43fcf42>] ? common_interrupt+0x82/0x82
[268342.187874]  <EOI> d [<ffffffffb42c23c3>] ? cpuidle_enter_state+0x113/0x260
[268342.187915]  [<ffffffffb3ebc00e>] ? cpu_startup_entry+0x17e/0x260
[268342.187952]  [<ffffffffb3e4845d>] ? start_secondary+0x14d/0x190
[268342.187989] Code: c48 cc7 cc1 c00 c93 c69 cb4 c48 cc7 cc7 cc9 ca7 c85 cb4 ce8 c4c cd8 cd4 cff c48 c83 cf8 c01 c19 cc0 c83 ce0 cf4 cc3 c66 c90 c0f c1f c44 c00 c00 cf6 c87 cc4 c02 c00 c00 c01 c<74> c45 c0f cb6
c87 c49 c01 c00 c00 c48 c89 cc2 c48 cc1 ce0 c08 c48 cc1 ce2 c04 c48 c

This in itself is already a Problem, but the following error made it even worse:

Jul 27 15:18:51 matthaes-web02 dockerd[1005]: time="2017-07-27T15:18:51.017058927+02:00" level=warning msg="Unable to locate plugin: pxd, retrying in 1s"
Jul 27 15:18:52 matthaes-web02 dockerd[1005]: time="2017-07-27T15:18:52.017399004+02:00" level=warning msg="Unable to locate plugin: pxd, retrying in 2s"
Jul 27 15:18:54 matthaes-web02 dockerd[1005]: time="2017-07-27T15:18:54.017747120+02:00" level=warning msg="Unable to locate plugin: pxd, retrying in 4s"

Docker does not start because Portworx is not available, which starts only after docker is up.
This is a deadlock and I was only able to solve it by removing /var/lib/docker on the hosts