I've created 3 vols in 3 different AZs: sa-east-1a, sa-east-b and sa-east-1c, each one with 30GB:
After that, I've created the ami and launched a new instance from it with this userdata:
Checking the AWS console, I've noticed that portworx created a lot of 30GB disks in all 3 zones!
This is the container log. It seems that it entered in a looping state:
Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac
Mon Jul 17 20:19:36 UTC 2017 : Running on Linux ip-10-0-1-73 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
checking /hostusr/src/kernels/4.4.0-83-generic
checking /hostusr/src/linux-headers-4.4.0-83-generic
checking /usr/src/kernels/4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Flushing buffer-cache..
checking /usr/src/linux-headers-4.4.0-83-generic
found /usr/src/linux-headers-4.4.0-83-generic
Creating px fs...
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Using storage device: vol-058d9c98020c8d4c9
Using storage device: vol-002334b285798b4ac
Fingerprints values:
HOSTNAME={"type": 1, "value": "ip-10-0-1-73"}
DISK={ "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }
NETWORK={ "type": 7, "value": "" }
NETWORK={ "type": 8, "value": "" }
FP_FINGERPRINT="Fingerprint": [{"type": 1, "value": "ip-10-0-1-73"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz" }, { "type": 5, "value": "3852904 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }]
FP_CONFIG={ "token": "", "create_profile": true, "profile_name": "ip-10-0-1-73-NoqPxviZ", "Fingerprint": [{"type": 1, "value": "ip-10-0-1-73"}, { "type": 2, "value": "Unknown_Vendor" }, { "type": 3, "value": "Unknown_Model" }, { "type": 4, "value": "Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz" }, { "type": 5, "value": "3852904 kB" }, { "type": 6, "value": "vol-0842059ddbdbd9d7f" }, { "type": 6, "value": "vol-058d9c98020c8d4c9" }, { "type": 6, "value": "vol-002334b285798b4ac" }, { "type": 7, "value": "" }, { "type": 8, "value": "" }] }
To use the same hardware configuration on an identical node, please use hardware profile ip-10-0-1-73-NoqPxviZ.
example:
curl -fsSL http://lighthouse.portworx.com/px_bootstrap.sh > px_bootstrap.sh; chmod u+x px_bootstrap.sh; PWX_TOKEN= PWX_CLUSTERNAME=socialbase bootstrap.sh -p ip-10-0-1-73-NoqPxviZ
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Failed to enable rt scheduler
Checking sysfs mount...
Executing with arguments: -c socialbase -k consul:http://swarmmanager.sb:8500 -s vol-0842059ddbdbd9d7f -s vol-058d9c98020c8d4c9 -s vol-002334b285798b4ac
Mon Jul 17 20:20:03 UTC 2017 : Running on Linux ip-10-0-1-73 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
size for /dev/shm is 67108864, less than required 293601280
PXD version: 33cafba6c39c862340a8c30b2677849f67bd2d6a
Using cluster: socialbase
Key Value Store: consul:http://swarmmanager.sb:8500
Using storage device: vol-0842059ddbdbd9d7f
Using storage device: vol-058d9c98020c8d4c9
Using storage device: vol-002334b285798b4ac
/docker-entry-point.sh: line 763: /sys/fs/cgroup/cpu/cpu.rt_runtime_us: Permission denied
Failed to enable rt scheduler
Checking sysfs mount...
2017-07-17 20:20:04,943 CRIT Supervisor running as root (no user in config file)
2017-07-17 20:20:04,947 INFO supervisord started with pid 1
2017-07-17 20:20:05,949 INFO spawned: 'relayd' with pid 75
2017-07-17 20:20:05,950 INFO spawned: 'lttng' with pid 76
2017-07-17 20:20:05,952 INFO spawned: 'exec' with pid 77
2017-07-17 20:20:05,953 INFO spawned: 'pxdaemon' with pid 78
2017-07-17 20:20:05,954 INFO spawned: 'px-ns' with pid 79
2017-07-17 20:20:05,956 INFO spawned: 'px_event_listener' with pid 80
PXPROCS: lttng not started yet...sleeping
Mon Jul 17 20:20:06 UTC 2017 cleaning up old traces
Mon Jul 17 20:20:06 UTC 2017 done cleaning up old traces
time="2017-07-17T20:20:07Z" level=info msg="px-ns Starting.."
2017-07-17 20:20:07,139 INFO success: relayd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,139 INFO success: lttng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: px-ns entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:20:07,140 INFO success: px_event_listener entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
NS client starting fuse module
Starting NS server
PXPROCS: lttng not started yet...sleeping
PXPROCS: lttng not started yet...sleeping
PXPROCS: lttng not started yet...sleeping
Spawning a session daemon
Session pxd created.
Traces will be written in net://localhost
Live timer set to 2000000 usec
Session pxd set to shm_path: /var/lib/osd/lttng/pxd-20170717-202016.
UST channel pxd_channel enabled for session pxd
All UST events are enabled in channel pxd_channel
Tracing started for session pxd
PXPROCS: Started px-storage with pid 152
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
PXPROCS: Started px with pid 163
PXPROCS: Started watchdog with pid 164
2017-07-17_20:20:21: PX-Watchdog: Starting watcher
2017-07-17_20:20:22: PX-Watchdog: Waiting for px process to start
root 163 78 0 20:20 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:20:23: PX-Watchdog: (pid 163): Begin monitoring
time="2017-07-17T20:20:28Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:20:28Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:20:29Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 1b35b6ce-8d86-41dc-9636-373f359a3af9). Trying attachSet..."
time="2017-07-17T20:20:29Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:20:29Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:20:29Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:20:29Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:20:29Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:20:29Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:20:29Z" level=info msg="New NodeID is db74ffaa-5541-4d7e-a78d-c44f709ecc97"
time="2017-07-17T20:20:29Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:20:29Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:20:29Z" level=info msg="Node db74ffaa-5541-4d7e-a78d-c44f709ecc97 with Index (184) is Up"
time="2017-07-17T20:20:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d5a0524e-b2a6-4d98-a0f7-f31bac8b3f78). Trying attachSet..."
time="2017-07-17T20:20:47Z" level=info msg="Failed to attach volume set vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..."
time="2017-07-17T20:20:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d7c87f0b-0833-4d22-a357-001ee84188fc). Trying attachSet..."
time="2017-07-17T20:20:50Z" level=info msg="Failed to attach volume set vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..."
time="2017-07-17T20:20:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00b0fb36e769745ad' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 944f071b-d693-4b5a-9f55-8d09132194ec). Trying attachSet..."
time="2017-07-17T20:20:53Z" level=info msg="Failed to attach volume set vol-0a1c8c9f619baf16e vol-00b0fb36e769745ad vol-0dc72eda0ec3c725a with error: No sets available. Retrying..."
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-0a1c8c9f619baf16e"
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-00b0fb36e769745ad"
time="2017-07-17T20:20:57Z" level=info msg="Deleted volume vol-0dc72eda0ec3c725a"
time="2017-07-17T20:20:57Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
152
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:20:58,301 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:20:59,303 INFO spawned: 'pxdaemon' with pid 214
PXPROCS: Started px-storage with pid 218
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:21:00,696 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 239
PXPROCS: Started watchdog with pid 240
2017-07-17_20:21:02: PX-Watchdog: Starting watcher
2017-07-17_20:21:02: PX-Watchdog: Waiting for px process to start
root 239 214 0 20:21 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:21:02: PX-Watchdog: (pid 239): Begin monitoring
time="2017-07-17T20:21:02Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:21:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:21:03Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 112b36fa-e59e-44f9-83cc-9c76515e5bc7). Trying attachSet..."
time="2017-07-17T20:21:03Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:21:03Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:21:03Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:21:03Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:21:03Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:21:03Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:21:03Z" level=info msg="New NodeID is f838f2b1-5e9c-45a0-a8ca-ec3d3f24bb00"
time="2017-07-17T20:21:03Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:21:03Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:21:03Z" level=info msg="Node f838f2b1-5e9c-45a0-a8ca-ec3d3f24bb00 with Index (185) is Up"
time="2017-07-17T20:21:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d2d87553-9173-4c45-b459-6819b68ead08). Trying attachSet..."
time="2017-07-17T20:21:17Z" level=info msg="Failed to attach volume set vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..."
time="2017-07-17T20:21:21Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8633021a-561b-452c-aead-2e824e743193). Trying attachSet..."
time="2017-07-17T20:21:21Z" level=info msg="Failed to attach volume set vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..."
time="2017-07-17T20:21:24Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00750f1049532859a' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d53417a1-eb9f-42bc-8132-5161445eae3c). Trying attachSet..."
time="2017-07-17T20:21:24Z" level=info msg="Failed to attach volume set vol-00750f1049532859a vol-02f7cbe7bcbafa260 vol-0d9c96794a8ddfd40 with error: No sets available. Retrying..."
2017-07-17 20:21:26,693 INFO reaped unknown pid 200
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-00750f1049532859a"
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-02f7cbe7bcbafa260"
time="2017-07-17T20:21:27Z" level=info msg="Deleted volume vol-0d9c96794a8ddfd40"
time="2017-07-17T20:21:27Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
218
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:21:28,702 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:21:29,705 INFO spawned: 'pxdaemon' with pid 323
PXPROCS: Started px-storage with pid 324
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:21:31,346 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 333
PXPROCS: Started watchdog with pid 334
2017-07-17_20:21:32: PX-Watchdog: Starting watcher
2017-07-17_20:21:32: PX-Watchdog: Waiting for px process to start
root 333 323 0 20:21 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:21:32: PX-Watchdog: (pid 333): Begin monitoring
time="2017-07-17T20:21:33Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:21:33Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:21:34Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4996b685-5ba7-4937-9c9d-aff07f2ba1a5). Trying attachSet..."
time="2017-07-17T20:21:34Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:21:34Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:21:34Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:21:34Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:21:34Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:21:34Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:21:34Z" level=info msg="New NodeID is e3f28af0-422e-4cb3-a4bf-877932e25ebb"
time="2017-07-17T20:21:34Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:21:34Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:21:34Z" level=info msg="Node e3f28af0-422e-4cb3-a4bf-877932e25ebb with Index (186) is Up"
time="2017-07-17T20:21:51Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 886f626e-2a8f-43a8-a048-6e5903792217). Trying attachSet..."
time="2017-07-17T20:21:51Z" level=info msg="Failed to attach volume set vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..."
time="2017-07-17T20:21:54Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6b193abd-1eb6-4179-af56-c2233c08b060). Trying attachSet..."
time="2017-07-17T20:21:54Z" level=info msg="Failed to attach volume set vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..."
time="2017-07-17T20:21:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-07bef9c07f4ca1bb7' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: e6841a93-07c8-483f-9439-91a51e1d6532). Trying attachSet..."
time="2017-07-17T20:21:58Z" level=info msg="Failed to attach volume set vol-07bef9c07f4ca1bb7 vol-0c484dcc58221cbee vol-0cae8c2f29ba14819 with error: No sets available. Retrying..."
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-07bef9c07f4ca1bb7"
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-0c484dcc58221cbee"
time="2017-07-17T20:22:01Z" level=info msg="Deleted volume vol-0cae8c2f29ba14819"
time="2017-07-17T20:22:01Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
324
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:22:02,681 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:22:03,683 INFO spawned: 'pxdaemon' with pid 384
PXPROCS: Started px-storage with pid 385
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:22:05,318 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:22:05,381 INFO reaped unknown pid 285
PXPROCS: Started px with pid 394
PXPROCS: Started watchdog with pid 395
2017-07-17_20:22:06: PX-Watchdog: Starting watcher
2017-07-17_20:22:06: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid218-time1500322887
2017-07-17_20:22:06: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:21:27.956992962 +0000 UTC.stack
2017-07-17_20:22:06: PX-Watchdog: Waiting for px process to start
root 394 384 0 20:22 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:22:06: PX-Watchdog: (pid 394): Begin monitoring
time="2017-07-17T20:22:06Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:22:06Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:22:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 04a1c231-6237-43e4-99f0-2a18b14e0409). Trying attachSet..."
time="2017-07-17T20:22:07Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:22:07Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:22:07Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:22:07Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:22:07Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:22:07Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:22:07Z" level=info msg="New NodeID is 1e53450a-d320-47f3-82e8-e3e38cfad2f6"
time="2017-07-17T20:22:07Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:22:07Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:22:07Z" level=info msg="Node 1e53450a-d320-47f3-82e8-e3e38cfad2f6 with Index (187) is Up"
Waiting for state transition to "attached".
time="2017-07-17T20:22:24Z" level=warning msg="Rollback attach volume /dev/xvdf, Error InvalidVolume.ZoneMismatch: The volume 'vol-0754cdc54af687106' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 06831c87-fd63-4363-a7a1-04b4e04021be"
time="2017-07-17T20:22:24Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0754cdc54af687106' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 06831c87-fd63-4363-a7a1-04b4e04021be). Trying attachSet..."
time="2017-07-17T20:22:24Z" level=info msg="Failed to attach volume set vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..."
time="2017-07-17T20:22:27Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-03fb2827619da2903 is already attached to an instance\n\tstatus code: 400, request id: 415096f6-2eb7-49f0-b255-be4807792aa2). Trying attachSet..."
time="2017-07-17T20:22:28Z" level=info msg="Failed to attach volume set vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..."
time="2017-07-17T20:22:31Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-03fb2827619da2903 is already attached to an instance\n\tstatus code: 400, request id: 026f0c84-b798-4a6e-884d-20cfbdc2e26e). Trying attachSet..."
time="2017-07-17T20:22:31Z" level=info msg="Failed to attach volume set vol-0754cdc54af687106 vol-0e9f2af07873a2459 vol-03fb2827619da2903 with error: No sets available. Retrying..."
time="2017-07-17T20:22:34Z" level=info msg="Deleted volume vol-0754cdc54af687106"
time="2017-07-17T20:22:35Z" level=info msg="Deleted volume vol-0e9f2af07873a2459"
time="2017-07-17T20:22:35Z" level=warning msg="Delete failed for volume vol-03fb2827619da2903, Error VolumeInUse: Volume vol-03fb2827619da2903 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 034beb8e-1b61-459b-8c8b-efb169ff6e46"
time="2017-07-17T20:22:35Z" level=info msg="Deleted volume vol-03fb2827619da2903"
time="2017-07-17T20:22:35Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
385
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:22:35,799 INFO reaped unknown pid 376
2017-07-17 20:22:35,833 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:22:36,835 INFO spawned: 'pxdaemon' with pid 451
PXPROCS: Started px-storage with pid 452
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:22:38,470 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 461
PXPROCS: Started watchdog with pid 462
2017-07-17_20:22:39: PX-Watchdog: Starting watcher
2017-07-17_20:22:39: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid324-time1500322921
2017-07-17_20:22:40: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:22:01.687706644 +0000 UTC.stack
2017-07-17_20:22:40: PX-Watchdog: Waiting for px process to start
root 461 451 14 20:22 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:22:40: PX-Watchdog: (pid 461): Begin monitoring
time="2017-07-17T20:22:40Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:22:40Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:22:40Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 70ef01ce-5998-4e22-bb2c-6a42a851e9f4). Trying attachSet..."
time="2017-07-17T20:22:40Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:22:40Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:22:40Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:22:40Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:22:40Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:22:40Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:22:40Z" level=info msg="New NodeID is aaf41dec-5162-48e0-a638-2090a13836e3"
time="2017-07-17T20:22:40Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:22:40Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:22:40Z" level=info msg="Node aaf41dec-5162-48e0-a638-2090a13836e3 with Index (188) is Up"
Waiting for state transition to "attached".
time="2017-07-17T20:22:57Z" level=warning msg="Rollback attach volume /dev/xvdg, Error InvalidVolume.ZoneMismatch: The volume 'vol-05fea0bb3a91338e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a61b135c-31b4-4c28-b5e9-5fce6af98684"
time="2017-07-17T20:22:57Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-05fea0bb3a91338e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a61b135c-31b4-4c28-b5e9-5fce6af98684). Trying attachSet..."
time="2017-07-17T20:22:58Z" level=info msg="Failed to attach volume set vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..."
time="2017-07-17T20:23:01Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-039e4ec32142653e2 is already attached to an instance\n\tstatus code: 400, request id: d6b02a83-9411-4750-8147-61c7cbd014f8). Trying attachSet..."
time="2017-07-17T20:23:01Z" level=info msg="Failed to attach volume set vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..."
time="2017-07-17T20:23:05Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-039e4ec32142653e2 is already attached to an instance\n\tstatus code: 400, request id: a894b483-2b4e-4fd5-beda-13f96b58ea91). Trying attachSet..."
time="2017-07-17T20:23:05Z" level=info msg="Failed to attach volume set vol-0c2263b7765a2b50b vol-05fea0bb3a91338e3 vol-039e4ec32142653e2 with error: No sets available. Retrying..."
time="2017-07-17T20:23:08Z" level=info msg="Deleted volume vol-0c2263b7765a2b50b"
time="2017-07-17T20:23:09Z" level=info msg="Deleted volume vol-05fea0bb3a91338e3"
time="2017-07-17T20:23:09Z" level=warning msg="Delete failed for volume vol-039e4ec32142653e2, Error VolumeInUse: Volume vol-039e4ec32142653e2 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: af83fcb4-4f42-4532-b6a7-5dd3cd8e453b"
time="2017-07-17T20:23:09Z" level=info msg="Deleted volume vol-039e4ec32142653e2"
time="2017-07-17T20:23:09Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
452
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:23:09,908 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:23:09,931 INFO spawned: 'pxdaemon' with pid 518
2017-07-17 20:23:09,932 INFO reaped unknown pid 442
PXPROCS: Started px-storage with pid 519
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:23:11,565 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 528
PXPROCS: Started watchdog with pid 529
2017-07-17_20:23:12: PX-Watchdog: Starting watcher
2017-07-17_20:23:13: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid385-time1500322955
2017-07-17_20:23:13: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:22:35.164295909 +0000 UTC.stack
2017-07-17_20:23:13: PX-Watchdog: Waiting for px process to start
root 528 518 0 20:23 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:23:13: PX-Watchdog: (pid 528): Begin monitoring
time="2017-07-17T20:23:13Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:23:13Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:23:13Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc68d073-1e09-4809-8d86-3afafba43fc1). Trying attachSet..."
time="2017-07-17T20:23:13Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:23:13Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:23:13Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:23:13Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:23:13Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:23:13Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:23:13Z" level=info msg="New NodeID is 0d56d20b-4c84-4505-a52e-b7faa353e301"
time="2017-07-17T20:23:13Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:23:13Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:23:13Z" level=info msg="Node 0d56d20b-4c84-4505-a52e-b7faa353e301 with Index (189) is Up"
time="2017-07-17T20:23:30Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 1b982094-f18f-454e-8bfd-0e8c410140a9). Trying attachSet..."
time="2017-07-17T20:23:31Z" level=info msg="Failed to attach volume set vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..."
time="2017-07-17T20:23:34Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c15f63b7-e38a-4f33-a933-4b44914d3081). Trying attachSet..."
time="2017-07-17T20:23:34Z" level=info msg="Failed to attach volume set vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..."
time="2017-07-17T20:23:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03ce75f45bd0bc5e6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 383087a1-63f7-40c6-903a-daaacf5ad088). Trying attachSet..."
time="2017-07-17T20:23:38Z" level=info msg="Failed to attach volume set vol-0e0355738c9daf989 vol-03ce75f45bd0bc5e6 vol-0b4104817dc18bb25 with error: No sets available. Retrying..."
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-0e0355738c9daf989"
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-03ce75f45bd0bc5e6"
time="2017-07-17T20:23:41Z" level=info msg="Deleted volume vol-0b4104817dc18bb25"
time="2017-07-17T20:23:41Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
519
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:23:42,424 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:23:43,056 INFO spawned: 'pxdaemon' with pid 583
2017-07-17 20:23:43,057 INFO reaped unknown pid 509
PXPROCS: Started px-storage with pid 584
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:23:44,678 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 593
PXPROCS: Started watchdog with pid 594
2017-07-17_20:23:46: PX-Watchdog: Starting watcher
2017-07-17_20:23:46: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid452-time1500322989
2017-07-17_20:23:46: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:23:09.155729831 +0000 UTC.stack
2017-07-17_20:23:46: PX-Watchdog: Waiting for px process to start
root 593 583 0 20:23 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:23:46: PX-Watchdog: (pid 593): Begin monitoring
time="2017-07-17T20:23:46Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:23:46Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:23:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 205a8bb7-2ea4-42ad-8b07-c83138bcb455). Trying attachSet..."
time="2017-07-17T20:23:47Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:23:47Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:23:47Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:23:47Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:23:47Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:23:47Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:23:47Z" level=info msg="New NodeID is 43b0f079-e184-47fa-a57e-3ddcefacc686"
time="2017-07-17T20:23:47Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:23:47Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:23:47Z" level=info msg="Node 43b0f079-e184-47fa-a57e-3ddcefacc686 with Index (190) is Up"
time="2017-07-17T20:24:00Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d6383b44-e221-4625-88ae-0f4d675ce49e). Trying attachSet..."
time="2017-07-17T20:24:01Z" level=info msg="Failed to attach volume set vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..."
time="2017-07-17T20:24:04Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b7bf557a-6236-4aa3-9e8e-1aaecf7be7cc). Trying attachSet..."
time="2017-07-17T20:24:04Z" level=info msg="Failed to attach volume set vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..."
time="2017-07-17T20:24:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-021ace0cd6005e01d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ff8f6a0-5333-4023-894e-cd9541c4cc6d). Trying attachSet..."
time="2017-07-17T20:24:07Z" level=info msg="Failed to attach volume set vol-0602b317886ec4938 vol-021ace0cd6005e01d vol-073cc4cd01ab0067a with error: No sets available. Retrying..."
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-0602b317886ec4938"
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-021ace0cd6005e01d"
time="2017-07-17T20:24:11Z" level=info msg="Deleted volume vol-073cc4cd01ab0067a"
time="2017-07-17T20:24:11Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
584
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:24:12,374 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:24:13,377 INFO spawned: 'pxdaemon' with pid 647
PXPROCS: Started px-storage with pid 648
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:24:15,018 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:24:16,116 INFO reaped unknown pid 576
PXPROCS: Started px with pid 657
PXPROCS: Started watchdog with pid 658
2017-07-17_20:24:16: PX-Watchdog: Starting watcher
2017-07-17_20:24:16: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid519-time1500323021
2017-07-17_20:24:16: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:23:41.620964623 +0000 UTC.stack
2017-07-17_20:24:16: PX-Watchdog: Waiting for px process to start
root 657 647 0 20:24 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:24:16: PX-Watchdog: (pid 657): Begin monitoring
time="2017-07-17T20:24:16Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:24:16Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:24:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ff2462f-adad-4d28-a4df-7ebc3abad72f). Trying attachSet..."
time="2017-07-17T20:24:17Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:24:17Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:24:17Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:24:17Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:24:17Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:24:17Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:24:17Z" level=info msg="New NodeID is 80a52d5c-9f0b-4d91-a55d-0100a0d21543"
time="2017-07-17T20:24:17Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:24:17Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:24:17Z" level=info msg="Node 80a52d5c-9f0b-4d91-a55d-0100a0d21543 with Index (191) is Up"
time="2017-07-17T20:24:28Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d18194c7-c97c-4653-b74f-c4c327ddf09a). Trying attachSet..."
time="2017-07-17T20:24:28Z" level=info msg="Failed to attach volume set vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..."
time="2017-07-17T20:24:31Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bc0b728a-09ef-4cac-b2e8-693c940c9969). Trying attachSet..."
time="2017-07-17T20:24:31Z" level=info msg="Failed to attach volume set vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..."
time="2017-07-17T20:24:35Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-041b6e029a322b566' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: d926c609-e296-47d4-9b71-ef1a3badae7f). Trying attachSet..."
time="2017-07-17T20:24:35Z" level=info msg="Failed to attach volume set vol-041b6e029a322b566 vol-053f068cbfcb41ee8 vol-047bb73b87e28b7a4 with error: No sets available. Retrying..."
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-041b6e029a322b566"
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-053f068cbfcb41ee8"
time="2017-07-17T20:24:38Z" level=info msg="Deleted volume vol-047bb73b87e28b7a4"
time="2017-07-17T20:24:38Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
648
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:24:39,561 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:24:40,563 INFO spawned: 'pxdaemon' with pid 715
PXPROCS: Started px-storage with pid 716
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:24:42,172 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 725
PXPROCS: Started watchdog with pid 726
2017-07-17_20:24:43: PX-Watchdog: Starting watcher
2017-07-17_20:24:43: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid584-time1500323051
2017-07-17_20:24:43: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:24:11.35037889 +0000 UTC.stack
2017-07-17_20:24:43: PX-Watchdog: Waiting for px process to start
root 725 715 0 20:24 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:24:43: PX-Watchdog: (pid 725): Begin monitoring
time="2017-07-17T20:24:43Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:24:43Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:24:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 05156c82-e040-4670-977b-8ac6d28b5b2f). Trying attachSet..."
time="2017-07-17T20:24:44Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:24:44Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:24:44Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:24:44Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:24:44Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:24:44Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:24:44Z" level=info msg="New NodeID is e7a373e2-8745-4a34-a0b7-5d874ab16358"
time="2017-07-17T20:24:44Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:24:44Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:24:44Z" level=info msg="Node e7a373e2-8745-4a34-a0b7-5d874ab16358 with Index (192) is Up"
2017-07-17 20:24:49,228 INFO reaped unknown pid 641
time="2017-07-17T20:24:55Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6cdef988-ff36-4089-91de-0a7d31d36a06). Trying attachSet..."
time="2017-07-17T20:24:55Z" level=info msg="Failed to attach volume set vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..."
time="2017-07-17T20:24:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6fbd4955-6ead-4297-9394-8023998e90a1). Trying attachSet..."
time="2017-07-17T20:24:58Z" level=info msg="Failed to attach volume set vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..."
time="2017-07-17T20:25:02Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0657b54d2782a7bc6' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6799674c-d090-4f39-b2d4-8d4f26a89424). Trying attachSet..."
time="2017-07-17T20:25:02Z" level=info msg="Failed to attach volume set vol-0657b54d2782a7bc6 vol-0bb70306e52984a56 vol-0d703b168c35f282d with error: No sets available. Retrying..."
time="2017-07-17T20:25:05Z" level=info msg="Deleted volume vol-0657b54d2782a7bc6"
time="2017-07-17T20:25:05Z" level=info msg="Deleted volume vol-0bb70306e52984a56"
time="2017-07-17T20:25:06Z" level=info msg="Deleted volume vol-0d703b168c35f282d"
time="2017-07-17T20:25:06Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
716
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:25:06,799 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:25:07,802 INFO spawned: 'pxdaemon' with pid 779
PXPROCS: Started px-storage with pid 780
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:25:09,410 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 789
PXPROCS: Started watchdog with pid 790
2017-07-17_20:25:10: PX-Watchdog: Starting watcher
2017-07-17_20:25:10: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid648-time1500323078
2017-07-17_20:25:10: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:24:38.685498093 +0000 UTC.stack
2017-07-17_20:25:10: PX-Watchdog: Waiting for px process to start
root 789 779 8 20:25 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:25:10: PX-Watchdog: (pid 789): Begin monitoring
time="2017-07-17T20:25:11Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:25:11Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:25:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4b606935-7d8b-45ed-b606-404ad255af68). Trying attachSet..."
time="2017-07-17T20:25:11Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:25:11Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:25:11Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:25:11Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:25:11Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:25:11Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:25:11Z" level=info msg="New NodeID is 10777ebd-50b0-4fa5-be24-fe3f1853350c"
time="2017-07-17T20:25:11Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:25:11Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:25:11Z" level=info msg="Node 10777ebd-50b0-4fa5-be24-fe3f1853350c with Index (193) is Up"
2017-07-17 20:25:19,526 INFO reaped unknown pid 705
time="2017-07-17T20:25:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 03968f47-0d51-4711-b4bc-299d89bceadb). Trying attachSet..."
time="2017-07-17T20:25:22Z" level=info msg="Failed to attach volume set vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..."
time="2017-07-17T20:25:25Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c8964c28-6a18-44ad-9d9e-8a981987690a). Trying attachSet..."
time="2017-07-17T20:25:26Z" level=info msg="Failed to attach volume set vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..."
time="2017-07-17T20:25:29Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0bc3d3459e618fd8b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 33939c5f-7e61-4c5a-9efd-adaa5fb93181). Trying attachSet..."
time="2017-07-17T20:25:29Z" level=info msg="Failed to attach volume set vol-0bc3d3459e618fd8b vol-0ccd3d4f8966fd354 vol-0f101dfd08e86c2c5 with error: No sets available. Retrying..."
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0bc3d3459e618fd8b"
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0ccd3d4f8966fd354"
time="2017-07-17T20:25:32Z" level=info msg="Deleted volume vol-0f101dfd08e86c2c5"
time="2017-07-17T20:25:32Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
780
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:25:33,524 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:25:34,527 INFO spawned: 'pxdaemon' with pid 844
PXPROCS: Started px-storage with pid 845
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:25:36,158 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 854
PXPROCS: Started watchdog with pid 855
2017-07-17_20:25:37: PX-Watchdog: Starting watcher
2017-07-17_20:25:37: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid716-time1500323106
2017-07-17_20:25:37: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:25:06.079558835 +0000 UTC.stack
2017-07-17_20:25:37: PX-Watchdog: Waiting for px process to start
root 854 844 0 20:25 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:25:37: PX-Watchdog: (pid 854): Begin monitoring
time="2017-07-17T20:25:37Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:25:37Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:25:38Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 49eb1ddd-35e3-423c-a54d-14821585fa39). Trying attachSet..."
time="2017-07-17T20:25:38Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:25:38Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:25:38Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:25:38Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:25:38Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:25:38Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:25:38Z" level=info msg="New NodeID is 1cb08f7e-ec86-41de-a2f9-6adc19314511"
time="2017-07-17T20:25:38Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:25:38Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:25:38Z" level=info msg="Node 1cb08f7e-ec86-41de-a2f9-6adc19314511 with Index (194) is Up"
2017-07-17 20:25:46,728 INFO reaped unknown pid 773
time="2017-07-17T20:25:52Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4130e767-414f-4566-8a7c-04ad1f787ffb). Trying attachSet..."
time="2017-07-17T20:25:52Z" level=info msg="Failed to attach volume set vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..."
time="2017-07-17T20:25:56Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc67c369-235e-4463-aab4-f8ec7cd0d290). Trying attachSet..."
time="2017-07-17T20:25:56Z" level=info msg="Failed to attach volume set vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..."
time="2017-07-17T20:25:59Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01a5dd80d92e4d7b9' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 7d7e1823-c775-44b8-ba7f-e9ff9df45a9f). Trying attachSet..."
time="2017-07-17T20:25:59Z" level=info msg="Failed to attach volume set vol-01a5dd80d92e4d7b9 vol-0b4b020d6fa2b965f vol-080f3e66e286102cb with error: No sets available. Retrying..."
time="2017-07-17T20:26:02Z" level=info msg="Deleted volume vol-01a5dd80d92e4d7b9"
time="2017-07-17T20:26:02Z" level=info msg="Deleted volume vol-0b4b020d6fa2b965f"
time="2017-07-17T20:26:03Z" level=info msg="Deleted volume vol-080f3e66e286102cb"
time="2017-07-17T20:26:03Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
845
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:26:03,930 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:26:04,933 INFO spawned: 'pxdaemon' with pid 910
PXPROCS: Started px-storage with pid 911
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:26:06,583 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 920
PXPROCS: Started watchdog with pid 921
2017-07-17_20:26:07: PX-Watchdog: Starting watcher
2017-07-17_20:26:08: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid780-time1500323132
2017-07-17_20:26:08: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:25:32.845892519 +0000 UTC.stack
2017-07-17_20:26:08: PX-Watchdog: Waiting for px process to start
root 920 910 0 20:26 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:26:08: PX-Watchdog: (pid 920): Begin monitoring
time="2017-07-17T20:26:08Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:26:08Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:26:08Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8da904b6-3771-4a9a-a141-50377f617384). Trying attachSet..."
time="2017-07-17T20:26:08Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:26:08Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:26:08Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:26:08Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:26:09Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:26:09Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:26:09Z" level=info msg="New NodeID is f081adb9-2e28-4eae-819c-cac8a914ec5a"
time="2017-07-17T20:26:09Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:26:09Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:26:09Z" level=info msg="Node f081adb9-2e28-4eae-819c-cac8a914ec5a with Index (195) is Up"
2017-07-17 20:26:13,991 INFO reaped unknown pid 838
Waiting for state transition to "attached".
time="2017-07-17T20:26:22Z" level=warning msg="Rollback attach volume /dev/xvdh, Error InvalidVolume.ZoneMismatch: The volume 'vol-050a22aa7316ebf6c' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0544a301-a544-47a2-8355-51c49af6d571"
time="2017-07-17T20:26:22Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-050a22aa7316ebf6c' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0544a301-a544-47a2-8355-51c49af6d571). Trying attachSet..."
time="2017-07-17T20:26:22Z" level=info msg="Failed to attach volume set vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..."
time="2017-07-17T20:26:26Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01005566936b4286a is already attached to an instance\n\tstatus code: 400, request id: f4c9ed99-6e10-4b8c-8816-c9bc7ce24df0). Trying attachSet..."
time="2017-07-17T20:26:26Z" level=info msg="Failed to attach volume set vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..."
time="2017-07-17T20:26:29Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01005566936b4286a is already attached to an instance\n\tstatus code: 400, request id: 019de502-07d1-4eb9-916a-17447ee32a1b). Trying attachSet..."
time="2017-07-17T20:26:29Z" level=info msg="Failed to attach volume set vol-050a22aa7316ebf6c vol-0eb0a3b0133732862 vol-01005566936b4286a with error: No sets available. Retrying..."
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-050a22aa7316ebf6c"
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-0eb0a3b0133732862"
time="2017-07-17T20:26:33Z" level=warning msg="Delete failed for volume vol-01005566936b4286a, Error VolumeInUse: Volume vol-01005566936b4286a is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: c3786596-2d41-4cfe-b653-2907144b0cd9"
time="2017-07-17T20:26:33Z" level=info msg="Deleted volume vol-01005566936b4286a"
time="2017-07-17T20:26:33Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
911
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:26:34,013 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:26:35,017 INFO spawned: 'pxdaemon' with pid 978
PXPROCS: Started px-storage with pid 979
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:26:36,651 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 988
PXPROCS: Started watchdog with pid 989
2017-07-17_20:26:38: PX-Watchdog: Starting watcher
2017-07-17_20:26:38: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid845-time1500323163
2017-07-17_20:26:38: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:26:03.08213779 +0000 UTC.stack
2017-07-17_20:26:38: PX-Watchdog: Waiting for px process to start
root 988 978 0 20:26 ? 00:00:00 /usr/local/bin/px -daemon
time="2017-07-17T20:26:38Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:26:38Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
2017-07-17_20:26:38: PX-Watchdog: (pid 988): Begin monitoring
time="2017-07-17T20:26:38Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 042f48e1-4a74-4282-b62c-120f0b505275). Trying attachSet..."
time="2017-07-17T20:26:38Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:26:38Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:26:38Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:26:38Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:26:38Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:26:38Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:26:38Z" level=info msg="New NodeID is 7fa4353d-bdb0-407e-ad50-61009a93446e"
time="2017-07-17T20:26:38Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:26:38Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:26:38Z" level=info msg="Node 7fa4353d-bdb0-407e-ad50-61009a93446e with Index (196) is Up"
2017-07-17 20:26:40,727 INFO reaped unknown pid 902
time="2017-07-17T20:26:49Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 249eedb9-03d6-47bc-90e5-bac4a438d78d). Trying attachSet..."
time="2017-07-17T20:26:50Z" level=info msg="Failed to attach volume set vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..."
time="2017-07-17T20:26:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6732b0b1-4591-4653-b9a7-196cd8427385). Trying attachSet..."
time="2017-07-17T20:26:53Z" level=info msg="Failed to attach volume set vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..."
time="2017-07-17T20:26:56Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-099d813e8fd60f787' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 010aa708-329c-4477-91a7-309b61759a88). Trying attachSet..."
time="2017-07-17T20:26:56Z" level=info msg="Failed to attach volume set vol-099d813e8fd60f787 vol-0ebc07b08c4cde4d3 vol-0ee606659f735d922 with error: No sets available. Retrying..."
time="2017-07-17T20:26:59Z" level=info msg="Deleted volume vol-099d813e8fd60f787"
time="2017-07-17T20:26:59Z" level=info msg="Deleted volume vol-0ebc07b08c4cde4d3"
time="2017-07-17T20:27:00Z" level=info msg="Deleted volume vol-0ee606659f735d922"
time="2017-07-17T20:27:00Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
979
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:27:01,063 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:02,066 INFO spawned: 'pxdaemon' with pid 1044
PXPROCS: Started px-storage with pid 1045
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:27:03,694 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1054
PXPROCS: Started watchdog with pid 1055
2017-07-17_20:27:05: PX-Watchdog: Starting watcher
2017-07-17_20:27:05: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid911-time1500323193
2017-07-17_20:27:05: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:26:33.397134213 +0000 UTC.stack
2017-07-17_20:27:05: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:27:05Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:27:05Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
root 1054 1044 0 20:27 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:27:05: PX-Watchdog: (pid 1054): Begin monitoring
time="2017-07-17T20:27:05Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: dafee0bb-631e-485b-8a8b-ba2b859a38a8). Trying attachSet..."
time="2017-07-17T20:27:06Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:27:06Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:27:06Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:27:06Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:27:06Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:27:06Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:27:06Z" level=info msg="New NodeID is 20d763d5-b44b-40fc-83f3-9976aa6090d6"
time="2017-07-17T20:27:06Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:27:06Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:27:06Z" level=info msg="Node 20d763d5-b44b-40fc-83f3-9976aa6090d6 with Index (197) is Up"
2017-07-17 20:27:11,081 INFO reaped unknown pid 968
time="2017-07-17T20:27:19Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 95a12bef-eb02-4fd1-b2d9-b457b5f38350). Trying attachSet..."
time="2017-07-17T20:27:20Z" level=info msg="Failed to attach volume set vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..."
time="2017-07-17T20:27:23Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2f0b7dfc-30f9-4d2a-a3cc-3155377b78b3). Trying attachSet..."
time="2017-07-17T20:27:23Z" level=info msg="Failed to attach volume set vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..."
time="2017-07-17T20:27:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-007d1c99f1baf09ab' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 9c1b5212-1047-4015-a9b4-717691e7f4a7). Trying attachSet..."
time="2017-07-17T20:27:26Z" level=info msg="Failed to attach volume set vol-011bc83c090da3408 vol-007d1c99f1baf09ab vol-0ba1e6a53ba00a031 with error: No sets available. Retrying..."
time="2017-07-17T20:27:29Z" level=info msg="Deleted volume vol-011bc83c090da3408"
time="2017-07-17T20:27:29Z" level=info msg="Deleted volume vol-007d1c99f1baf09ab"
time="2017-07-17T20:27:30Z" level=info msg="Deleted volume vol-0ba1e6a53ba00a031"
time="2017-07-17T20:27:30Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1045
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:27:30,813 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:31,816 INFO spawned: 'pxdaemon' with pid 1110
PXPROCS: Started px-storage with pid 1111
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:27:33,430 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1120
PXPROCS: Started watchdog with pid 1121
2017-07-17_20:27:34: PX-Watchdog: Starting watcher
2017-07-17_20:27:34: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid979-time1500323220
2017-07-17_20:27:34: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:00.237474507 +0000 UTC.stack
2017-07-17_20:27:35: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:27:35Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:27:35Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
root 1120 1110 13 20:27 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:27:35: PX-Watchdog: (pid 1120): Begin monitoring
time="2017-07-17T20:27:35Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a8c4a83c-cc21-4111-a26f-488d8ecdecb5). Trying attachSet..."
time="2017-07-17T20:27:35Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:27:35Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:27:35Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:27:35Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:27:35Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:27:35Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:27:35Z" level=info msg="New NodeID is af5717a7-6264-4758-995b-e72ec7f915cc"
time="2017-07-17T20:27:35Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:27:35Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:27:35Z" level=info msg="Node af5717a7-6264-4758-995b-e72ec7f915cc with Index (198) is Up"
2017-07-17 20:27:41,249 INFO reaped unknown pid 1039
time="2017-07-17T20:27:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 172e05e6-8748-4f0b-8011-5d1477c0441a). Trying attachSet..."
time="2017-07-17T20:27:46Z" level=info msg="Failed to attach volume set vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..."
time="2017-07-17T20:27:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: f2f608f7-c284-44df-b4d5-e50a71f9a46a). Trying attachSet..."
time="2017-07-17T20:27:50Z" level=info msg="Failed to attach volume set vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..."
time="2017-07-17T20:27:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0689dea182e8ae258' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: af94b5af-4b5b-4007-a3c4-21cbe37b7f01). Trying attachSet..."
time="2017-07-17T20:27:53Z" level=info msg="Failed to attach volume set vol-0689dea182e8ae258 vol-0d2f6ebed91295e99 vol-0a29d43b2e71318d8 with error: No sets available. Retrying..."
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0689dea182e8ae258"
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0d2f6ebed91295e99"
time="2017-07-17T20:27:56Z" level=info msg="Deleted volume vol-0a29d43b2e71318d8"
time="2017-07-17T20:27:56Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1111
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:27:57,701 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:27:58,703 INFO spawned: 'pxdaemon' with pid 1176
PXPROCS: Started px-storage with pid 1177
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:00,347 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1186
PXPROCS: Started watchdog with pid 1187
2017-07-17_20:28:01: PX-Watchdog: Starting watcher
2017-07-17_20:28:01: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1045-time1500323250
2017-07-17_20:28:01: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:30.154572756 +0000 UTC.stack
2017-07-17_20:28:01: PX-Watchdog: Waiting for px process to start
root 1186 1176 0 20:28 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:01: PX-Watchdog: (pid 1186): Begin monitoring
time="2017-07-17T20:28:01Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:28:01Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:28:02Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: f0c15705-922f-494a-89cd-84a4a543122d). Trying attachSet..."
time="2017-07-17T20:28:02Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:28:02Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:28:02Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:28:02Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:28:02Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:28:02Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:28:02Z" level=info msg="New NodeID is 45bb1f10-c864-4d1e-85d8-2164533074ed"
time="2017-07-17T20:28:02Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:28:02Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:28:02Z" level=info msg="Node 45bb1f10-c864-4d1e-85d8-2164533074ed with Index (199) is Up"
2017-07-17 20:28:08,335 INFO reaped unknown pid 1104
Waiting for state transition to "attached".
time="2017-07-17T20:28:16Z" level=warning msg="Rollback attach volume /dev/xvdi, Error InvalidVolume.ZoneMismatch: The volume 'vol-037a2b5f2a17e29a3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a0836ff2-f7a1-42f5-b671-56ce701090b9"
time="2017-07-17T20:28:16Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-037a2b5f2a17e29a3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a0836ff2-f7a1-42f5-b671-56ce701090b9). Trying attachSet..."
time="2017-07-17T20:28:16Z" level=info msg="Failed to attach volume set vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..."
time="2017-07-17T20:28:20Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01d3841fc605f322f is already attached to an instance\n\tstatus code: 400, request id: b07d643c-5963-4675-94c3-55a77a0ce75c). Trying attachSet..."
time="2017-07-17T20:28:20Z" level=info msg="Failed to attach volume set vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..."
time="2017-07-17T20:28:23Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-01d3841fc605f322f is already attached to an instance\n\tstatus code: 400, request id: af301d66-945c-44c1-a39c-0e60c957db56). Trying attachSet..."
time="2017-07-17T20:28:24Z" level=info msg="Failed to attach volume set vol-0bed0efe74e5c82e5 vol-037a2b5f2a17e29a3 vol-01d3841fc605f322f with error: No sets available. Retrying..."
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-0bed0efe74e5c82e5"
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-037a2b5f2a17e29a3"
time="2017-07-17T20:28:27Z" level=warning msg="Delete failed for volume vol-01d3841fc605f322f, Error VolumeInUse: Volume vol-01d3841fc605f322f is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: f7e4a11b-8f8e-4414-b052-9c2b48e6bce2"
time="2017-07-17T20:28:27Z" level=info msg="Deleted volume vol-01d3841fc605f322f"
time="2017-07-17T20:28:27Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1177
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:28:27,492 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:28:28,495 INFO spawned: 'pxdaemon' with pid 1245
PXPROCS: Started px-storage with pid 1246
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:30,130 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1255
PXPROCS: Started watchdog with pid 1256
2017-07-17_20:28:31: PX-Watchdog: Starting watcher
2017-07-17_20:28:31: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1111-time1500323276
2017-07-17_20:28:31: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:27:56.973406987 +0000 UTC.stack
2017-07-17_20:28:31: PX-Watchdog: Waiting for px process to start
root 1255 1245 0 20:28 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:31: PX-Watchdog: (pid 1255): Begin monitoring
time="2017-07-17T20:28:31Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:28:31Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:28:32Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 63e35a20-78cf-4f60-af8f-9124245cd108). Trying attachSet..."
time="2017-07-17T20:28:32Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:28:32Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:28:32Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:28:32Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:28:32Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:28:32Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:28:32Z" level=info msg="New NodeID is 77042558-cabb-4d05-877b-d2105e95088b"
time="2017-07-17T20:28:32Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:28:32Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:28:32Z" level=info msg="Node 77042558-cabb-4d05-877b-d2105e95088b with Index (200) is Up"
2017-07-17 20:28:38,064 INFO reaped unknown pid 1170
time="2017-07-17T20:28:43Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 307736dd-d7fc-462b-b35c-ff431f41c9e3). Trying attachSet..."
time="2017-07-17T20:28:43Z" level=info msg="Failed to attach volume set vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..."
time="2017-07-17T20:28:46Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 176322cd-3053-4c15-bd87-e4fe4f930de0). Trying attachSet..."
time="2017-07-17T20:28:47Z" level=info msg="Failed to attach volume set vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..."
time="2017-07-17T20:28:50Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-04a8c069f6593c478' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b20869bc-923c-4a0d-9812-2e56131c407f). Trying attachSet..."
time="2017-07-17T20:28:50Z" level=info msg="Failed to attach volume set vol-07bd9c31eee1a5369 vol-04a8c069f6593c478 vol-07b74ab9dd4c1516e with error: No sets available. Retrying..."
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-07bd9c31eee1a5369"
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-04a8c069f6593c478"
time="2017-07-17T20:28:53Z" level=info msg="Deleted volume vol-07b74ab9dd4c1516e"
time="2017-07-17T20:28:53Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1246
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:28:53,992 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:28:54,995 INFO spawned: 'pxdaemon' with pid 1311
PXPROCS: Started px-storage with pid 1312
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:28:56,611 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1321
PXPROCS: Started watchdog with pid 1322
2017-07-17_20:28:58: PX-Watchdog: Starting watcher
2017-07-17_20:28:58: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1177-time1500323307
2017-07-17_20:28:58: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:28:27.364753832 +0000 UTC.stack
2017-07-17_20:28:58: PX-Watchdog: Waiting for px process to start
time="2017-07-17T20:28:58Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:28:58Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
root 1321 1311 0 20:28 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:28:58: PX-Watchdog: (pid 1321): Begin monitoring
time="2017-07-17T20:28:58Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: ea863fee-bd89-4fbb-b64c-418dd0c8e652). Trying attachSet..."
time="2017-07-17T20:28:58Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:28:58Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:28:58Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:28:58Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:28:58Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:28:58Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:28:58Z" level=info msg="New NodeID is bcf1228f-b5c9-4bd9-a01b-525cf3e9df9f"
time="2017-07-17T20:28:58Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:28:58Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:28:58Z" level=info msg="Node bcf1228f-b5c9-4bd9-a01b-525cf3e9df9f with Index (201) is Up"
2017-07-17 20:29:04,856 INFO reaped unknown pid 1234
Waiting for state transition to "attached".
time="2017-07-17T20:29:12Z" level=warning msg="Rollback attach volume /dev/xvdj, Error InvalidVolume.ZoneMismatch: The volume 'vol-01da48b0445fed392' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 31c3903a-9e85-48ba-9121-74108bdb060b"
time="2017-07-17T20:29:12Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-01da48b0445fed392' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 31c3903a-9e85-48ba-9121-74108bdb060b). Trying attachSet..."
time="2017-07-17T20:29:12Z" level=info msg="Failed to attach volume set vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..."
time="2017-07-17T20:29:15Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00aa97e0ca04b783e is already attached to an instance\n\tstatus code: 400, request id: 5d08c26e-cd8b-4382-a14f-63e176371ea8). Trying attachSet..."
time="2017-07-17T20:29:16Z" level=info msg="Failed to attach volume set vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..."
time="2017-07-17T20:29:19Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00aa97e0ca04b783e is already attached to an instance\n\tstatus code: 400, request id: edb3fe8e-78ce-41f6-9482-b4f677b8a631). Trying attachSet..."
time="2017-07-17T20:29:19Z" level=info msg="Failed to attach volume set vol-0e600499b93e24abe vol-01da48b0445fed392 vol-00aa97e0ca04b783e with error: No sets available. Retrying..."
time="2017-07-17T20:29:22Z" level=info msg="Deleted volume vol-0e600499b93e24abe"
time="2017-07-17T20:29:23Z" level=info msg="Deleted volume vol-01da48b0445fed392"
time="2017-07-17T20:29:23Z" level=warning msg="Delete failed for volume vol-00aa97e0ca04b783e, Error VolumeInUse: Volume vol-00aa97e0ca04b783e is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 8b5a2444-114e-4cec-a746-a44d57e0d8bd"
time="2017-07-17T20:29:23Z" level=info msg="Deleted volume vol-00aa97e0ca04b783e"
time="2017-07-17T20:29:23Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1312
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:29:23,313 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:29:24,315 INFO spawned: 'pxdaemon' with pid 1376
PXPROCS: Started px-storage with pid 1377
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:29:25,971 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1386
PXPROCS: Started watchdog with pid 1387
2017-07-17_20:29:27: PX-Watchdog: Starting watcher
2017-07-17_20:29:27: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1246-time1500323333
2017-07-17_20:29:27: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:28:53.69988716 +0000 UTC.stack
2017-07-17_20:29:27: PX-Watchdog: Waiting for px process to start
root 1386 1376 0 20:29 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:29:27: PX-Watchdog: (pid 1386): Begin monitoring
time="2017-07-17T20:29:27Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:29:27Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:29:28Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 8c76aa23-2d41-47f5-a37f-6184fa1eee55). Trying attachSet..."
time="2017-07-17T20:29:28Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:29:28Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:29:28Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:29:28Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:29:28Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:29:28Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:29:28Z" level=info msg="New NodeID is d0fc3fea-3470-4ed2-9b9e-bb5d41443015"
time="2017-07-17T20:29:28Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:29:28Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:29:28Z" level=info msg="Node d0fc3fea-3470-4ed2-9b9e-bb5d41443015 with Index (202) is Up"
2017-07-17 20:29:34,668 INFO reaped unknown pid 1304
Waiting for state transition to "attached".
time="2017-07-17T20:29:47Z" level=warning msg="Rollback attach volume /dev/xvdk, Error InvalidVolume.ZoneMismatch: The volume 'vol-09ebb76decd2a21a5' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bbfccef8-85a2-41c8-9d91-bc200b9dff9e"
time="2017-07-17T20:29:47Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-09ebb76decd2a21a5' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: bbfccef8-85a2-41c8-9d91-bc200b9dff9e). Trying attachSet..."
time="2017-07-17T20:29:47Z" level=info msg="Failed to attach volume set vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..."
time="2017-07-17T20:29:51Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00a1007e4a44e2349 is already attached to an instance\n\tstatus code: 400, request id: 8f446eb2-bca8-408f-9d7b-90554efe480f). Trying attachSet..."
time="2017-07-17T20:29:51Z" level=info msg="Failed to attach volume set vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..."
time="2017-07-17T20:29:54Z" level=warning msg="AttachSet: attachVols failed with error (VolumeInUse: vol-00a1007e4a44e2349 is already attached to an instance\n\tstatus code: 400, request id: ccaf35d4-c687-485f-bac3-725f5690b3d7). Trying attachSet..."
time="2017-07-17T20:29:55Z" level=info msg="Failed to attach volume set vol-09ebb76decd2a21a5 vol-0c275e24e94030709 vol-00a1007e4a44e2349 with error: No sets available. Retrying..."
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-09ebb76decd2a21a5"
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-0c275e24e94030709"
time="2017-07-17T20:29:58Z" level=warning msg="Delete failed for volume vol-00a1007e4a44e2349, Error VolumeInUse: Volume vol-00a1007e4a44e2349 is currently attached to i-0aea7fee0e642911d\n\tstatus code: 400, request id: 3d4a0160-ad3e-41c6-adbd-d2d7bfa061b8"
time="2017-07-17T20:29:58Z" level=info msg="Deleted volume vol-00a1007e4a44e2349"
time="2017-07-17T20:29:58Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1377
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:29:58,749 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:29:59,752 INFO spawned: 'pxdaemon' with pid 1443
PXPROCS: Started px-storage with pid 1444
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:01,240 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-07-17 20:30:01,241 INFO reaped unknown pid 1369
PXPROCS: Started px with pid 1453
PXPROCS: Started watchdog with pid 1454
2017-07-17_20:30:02: PX-Watchdog: Starting watcher
2017-07-17_20:30:02: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1312-time1500323363
2017-07-17_20:30:02: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:29:23.190405658 +0000 UTC.stack
2017-07-17_20:30:02: PX-Watchdog: Waiting for px process to start
root 1453 1443 0 20:30 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:30:02: PX-Watchdog: (pid 1453): Begin monitoring
time="2017-07-17T20:30:02Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:30:02Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:30:03Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0ba5cfc3-15cb-4f96-90ba-fa2205c7b19c). Trying attachSet..."
time="2017-07-17T20:30:03Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:30:03Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:30:03Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:30:03Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:30:03Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:30:03Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:30:03Z" level=info msg="New NodeID is a0704554-ad8d-4e45-9233-0e5788efa64d"
time="2017-07-17T20:30:03Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:30:03Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:30:03Z" level=info msg="Node a0704554-ad8d-4e45-9233-0e5788efa64d with Index (203) is Up"
time="2017-07-17T20:30:15Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 5a88123a-1ebb-4c14-a4ac-b7af49a13e14). Trying attachSet..."
time="2017-07-17T20:30:15Z" level=info msg="Failed to attach volume set vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..."
time="2017-07-17T20:30:18Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: ca24dbfa-d0ed-4733-b4a4-d7ece3d88138). Trying attachSet..."
time="2017-07-17T20:30:18Z" level=info msg="Failed to attach volume set vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..."
time="2017-07-17T20:30:21Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002226758ec436a2b' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 4cc59be9-e927-4949-a7ce-2d0cbc25bfe5). Trying attachSet..."
time="2017-07-17T20:30:22Z" level=info msg="Failed to attach volume set vol-002226758ec436a2b vol-0c0c0d1b5f73d207c vol-0fddd4790e207a76e with error: No sets available. Retrying..."
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-002226758ec436a2b"
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-0c0c0d1b5f73d207c"
time="2017-07-17T20:30:25Z" level=info msg="Deleted volume vol-0fddd4790e207a76e"
time="2017-07-17T20:30:25Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1444
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:30:25,593 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:30:26,596 INFO spawned: 'pxdaemon' with pid 1510
PXPROCS: Started px-storage with pid 1511
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:28,238 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1520
PXPROCS: Started watchdog with pid 1521
2017-07-17_20:30:29: PX-Watchdog: Starting watcher
2017-07-17_20:30:29: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1377-time1500323398
2017-07-17_20:30:29: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:29:58.533024037 +0000 UTC.stack
2017-07-17_20:30:29: PX-Watchdog: Waiting for px process to start
root 1520 1510 0 20:30 ? 00:00:00 /usr/local/bin/px -daemon
time="2017-07-17T20:30:29Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:30:29Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
2017-07-17_20:30:29: PX-Watchdog: (pid 1520): Begin monitoring
2017-07-17 20:30:30,468 INFO reaped unknown pid 1434
time="2017-07-17T20:30:30Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2d90acf5-d8ed-417f-bdcd-02f668f7e8d2). Trying attachSet..."
time="2017-07-17T20:30:30Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:30:30Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:30:30Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:30:30Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:30:30Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:30:30Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:30:30Z" level=info msg="New NodeID is 1a26f325-6f17-4e8e-bf7e-3891de932909"
time="2017-07-17T20:30:30Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:30:30Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:30:30Z" level=info msg="Node 1a26f325-6f17-4e8e-bf7e-3891de932909 with Index (204) is Up"
time="2017-07-17T20:30:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fc8301d9-4931-4f69-8b17-97fa2468ace8). Trying attachSet..."
time="2017-07-17T20:30:45Z" level=info msg="Failed to attach volume set vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..."
time="2017-07-17T20:30:48Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 45effe32-fe2f-4d32-bfb2-f5d1c70a055e). Trying attachSet..."
time="2017-07-17T20:30:48Z" level=info msg="Failed to attach volume set vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..."
time="2017-07-17T20:30:51Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-03437c1d61237e06d' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0fa50a78-b1e6-4694-a420-1248e013a0a0). Trying attachSet..."
time="2017-07-17T20:30:51Z" level=info msg="Failed to attach volume set vol-03437c1d61237e06d vol-070c06c66a4edf5e3 vol-0def01de3c342c0b2 with error: No sets available. Retrying..."
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-03437c1d61237e06d"
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-070c06c66a4edf5e3"
time="2017-07-17T20:30:54Z" level=info msg="Deleted volume vol-0def01de3c342c0b2"
time="2017-07-17T20:30:54Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1511
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:30:55,281 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:30:56,283 INFO spawned: 'pxdaemon' with pid 1575
PXPROCS: Started px-storage with pid 1576
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:30:57,914 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1585
PXPROCS: Started watchdog with pid 1586
2017-07-17_20:30:59: PX-Watchdog: Starting watcher
2017-07-17_20:30:59: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1444-time1500323425
2017-07-17_20:30:59: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:30:25.457395001 +0000 UTC.stack
2017-07-17_20:30:59: PX-Watchdog: Waiting for px process to start
root 1585 1575 0 20:30 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:30:59: PX-Watchdog: (pid 1585): Begin monitoring
time="2017-07-17T20:30:59Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:30:59Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:31:00Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 0690e408-cb5e-4c52-873d-542b796400eb). Trying attachSet..."
time="2017-07-17T20:31:00Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:31:00Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:31:00Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:31:00Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:31:00Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:31:00Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:31:00Z" level=info msg="New NodeID is a62882c3-5db6-47dd-aa20-e5036752dd4d"
time="2017-07-17T20:31:00Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:31:00Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:31:00Z" level=info msg="Node a62882c3-5db6-47dd-aa20-e5036752dd4d with Index (205) is Up"
2017-07-17 20:31:05,929 INFO reaped unknown pid 1501
time="2017-07-17T20:31:10Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: b40dbeb9-7e09-4119-8d50-5335145f5edf). Trying attachSet..."
time="2017-07-17T20:31:11Z" level=info msg="Failed to attach volume set vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..."
time="2017-07-17T20:31:14Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 172f4a11-653c-4ce1-b2b7-79269b9fd632). Trying attachSet..."
time="2017-07-17T20:31:14Z" level=info msg="Failed to attach volume set vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..."
time="2017-07-17T20:31:17Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-0aafb49c766743134' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: c5e717fb-bb32-41bb-be69-326462c6e13c). Trying attachSet..."
time="2017-07-17T20:31:18Z" level=info msg="Failed to attach volume set vol-0aafb49c766743134 vol-0ff134bfb92c759c0 vol-0b1d81a6aeab63d89 with error: No sets available. Retrying..."
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0aafb49c766743134"
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0ff134bfb92c759c0"
time="2017-07-17T20:31:21Z" level=info msg="Deleted volume vol-0b1d81a6aeab63d89"
time="2017-07-17T20:31:21Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1576
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:31:21,813 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:31:22,815 INFO spawned: 'pxdaemon' with pid 1640
PXPROCS: Started px-storage with pid 1641
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:31:24,450 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1650
PXPROCS: Started watchdog with pid 1651
2017-07-17_20:31:25: PX-Watchdog: Starting watcher
2017-07-17_20:31:25: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1511-time1500323454
2017-07-17_20:31:26: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:30:54.91891449 +0000 UTC.stack
time="2017-07-17T20:31:26Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:31:26Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
2017-07-17_20:31:26: PX-Watchdog: Waiting for px process to start
root 1650 1640 15 20:31 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:31:26: PX-Watchdog: (pid 1650): Begin monitoring
time="2017-07-17T20:31:26Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 5259b7a5-046e-4165-82be-7f1424061637). Trying attachSet..."
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 2a0b4a24-27ed-4db9-887b-4b0f8695c57d - want 3 vols got 1"
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:31:26Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:31:26Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:31:26Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:31:26Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:31:26Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:31:26Z" level=info msg="New NodeID is 5e7942f5-cc34-41b3-a383-2002e46dc4c3"
time="2017-07-17T20:31:26Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:31:26Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:31:26Z" level=info msg="Node 5e7942f5-cc34-41b3-a383-2002e46dc4c3 with Index (206) is Up"
2017-07-17 20:31:32,844 INFO reaped unknown pid 1569
time="2017-07-17T20:31:37Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: e82c913d-2f17-429c-87dd-ea7471f7fa3f). Trying attachSet..."
time="2017-07-17T20:31:38Z" level=info msg="Failed to attach volume set vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..."
time="2017-07-17T20:31:41Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 2525bc39-3e6c-4bd1-8da6-8990ed2e197d). Trying attachSet..."
time="2017-07-17T20:31:41Z" level=info msg="Failed to attach volume set vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..."
time="2017-07-17T20:31:44Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-048e6ebce8493ec84' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: a913b333-7d5e-47bd-b034-4649afc28576). Trying attachSet..."
time="2017-07-17T20:31:44Z" level=info msg="Failed to attach volume set vol-0f4102b45575f73e8 vol-048e6ebce8493ec84 vol-0656f33ba1a241e3d with error: No sets available. Retrying..."
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-0f4102b45575f73e8"
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-048e6ebce8493ec84"
time="2017-07-17T20:31:48Z" level=info msg="Deleted volume vol-0656f33ba1a241e3d"
time="2017-07-17T20:31:48Z" level=warning msg="Unable to save new node configuration: No sets available"
PXPROCS: px daemon exited with code: 6
1641
PXPROCS: dumping core and killing px-storage...
2017-07-17 20:31:48,861 INFO exited: pxdaemon (exit status 6; not expected)
2017-07-17 20:31:49,863 INFO spawned: 'pxdaemon' with pid 1706
PXPROCS: Started px-storage with pid 1707
bash: connect: Connection refused
bash: /dev/tcp/localhost/9009: Connection refused
PXPROCS: px-storage not started yet...sleeping
C++ grpc server listening on 0.0.0.0:9009
2017-07-17 20:31:51,498 INFO success: pxdaemon entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
PXPROCS: Started px with pid 1716
PXPROCS: Started watchdog with pid 1717
2017-07-17_20:31:52: PX-Watchdog: Starting watcher
2017-07-17_20:31:52: PX-Watchdog: (pid 0): Removing intermediate px core file: core-px-storage-sig6-user0-group0-pid1576-time1500323481
2017-07-17_20:31:53: PX-Watchdog: (pid 0): Removing intermediate px core file: 2017-07-17 20:31:21.498835997 +0000 UTC.stack
2017-07-17_20:31:53: PX-Watchdog: Waiting for px process to start
root 1716 1706 10 20:31 ? 00:00:00 /usr/local/bin/px -daemon
2017-07-17_20:31:53: PX-Watchdog: (pid 1716): Begin monitoring
time="2017-07-17T20:31:53Z" level=info msg="Registering [kernel] as a volume driver"
time="2017-07-17T20:31:53Z" level=info msg="Starting PX Version: 1.2.8-e70082e - Build Version e70082e281be8b71872b09a3304926438466fc5b"
time="2017-07-17T20:31:53Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-002334b285798b4ac' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: fce46d5f-3bf4-4681-a1cb-84576b1d5916). Trying attachSet..."
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 07cbe42e-f3f6-4562-83d1-7e5bae1d1657 - want 3 vols got 1"
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 5adbf28b-6976-4a62-9275-05adccafb57a - want 3 vols got 1"
time="2017-07-17T20:31:53Z" level=warning msg="Skipping label 2a0b4a24-27ed-4db9-887b-4b0f8695c57d - want 3 vols got 1"
time="2017-07-17T20:31:53Z" level=info msg="AttachSet while Load of private config failed: Failed to find an available set"
time="2017-07-17T20:31:53Z" level=info msg="Node is not yet initialized"
time="2017-07-17T20:31:54Z" level=error msg="Unsupported scheduler none given to storage driver"
time="2017-07-17T20:31:54Z" level=info msg="Generating a new Node ID and index."
time="2017-07-17T20:31:54Z" level=info msg="New NodeID is a2832162-c916-452b-a9ec-056af976a982"
time="2017-07-17T20:31:54Z" level=info msg="Joining cluster socialbase at this discovery service: [swarmmanager.sb:8500]."
time="2017-07-17T20:31:54Z" level=info msg="Alerts initialized successfully for this cluster"
time="2017-07-17T20:31:54Z" level=info msg="Node a2832162-c916-452b-a9ec-056af976a982 with Index (207) is Up"
2017-07-17 20:32:02,471 INFO reaped unknown pid 1634
time="2017-07-17T20:32:07Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: dc7130f6-9f18-41d7-bbfe-c869ef99da05). Trying attachSet..."
time="2017-07-17T20:32:08Z" level=info msg="Failed to attach volume set vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..."
time="2017-07-17T20:32:11Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 6708a9f6-2fc0-4cc4-9311-7b988419744a). Trying attachSet..."
time="2017-07-17T20:32:11Z" level=info msg="Failed to attach volume set vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..."
time="2017-07-17T20:32:14Z" level=warning msg="AttachSet: attachVols failed with error (InvalidVolume.ZoneMismatch: The volume 'vol-00f8ce899fac130e3' is not in the same availability zone as instance 'i-0aea7fee0e642911d'\n\tstatus code: 400, request id: 86997d13-17d6-4fec-9c23-b6dea2d4248d). Trying attachSet..."
time="2017-07-17T20:32:14Z" level=info msg="Failed to attach volume set vol-049feacfbe7be8b0c vol-00f8ce899fac130e3 vol-0e0d5d89a61f6765a with error: No sets available. Retrying..."