Create service fails with timeout about cf-services-contrib-release

[jhiemer]

Next step, looking into gateways 0 I see:

root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/bosh_ssh/bosh_en9u1iqap# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.24.102.157:51794    0.0.0.0:*               LISTEN      17353/ruby      
tcp        0      0 127.0.0.1:33331         0.0.0.0:*               LISTEN      782/bosh-agent  
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1347/sshd       
tcp        0      0 0.0.0.0:57792           0.0.0.0:*               LISTEN      624/rpc.statd   
tcp        0      0 127.0.0.1:2822          0.0.0.0:*               LISTEN      1466/monit      
tcp        0      0 172.24.102.157:59401    0.0.0.0:*               LISTEN      17343/ruby      
tcp        0      0 127.0.0.1:2825          0.0.0.0:*               LISTEN      782/bosh-agent  
tcp        0      0 172.24.102.157:43722    0.0.0.0:*               LISTEN      17363/ruby      
tcp        0      0 172.24.102.157:50474    0.0.0.0:*               LISTEN      17333/ruby      
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      611/rpcbind     
tcp6       0      0 :::22                   :::*                    LISTEN      1347/sshd       
tcp6       0      0 :::36254                :::*                    LISTEN      624/rpc.statd   
tcp6       0      0 :::111                  :::*                    LISTEN      611/rpcbind     
udp        0      0 0.0.0.0:39395           0.0.0.0:*                           624/rpc.statd   
udp        0      0 0.0.0.0:111             0.0.0.0:*                           611/rpcbind     
udp        0      0 0.0.0.0:761             0.0.0.0:*                           611/rpcbind     
udp        0      0 127.0.0.1:800           0.0.0.0:*                           624/rpc.statd   
udp6       0      0 :::35329                :::*                                624/rpc.statd   
udp6       0      0 :::111                  :::*                                611/rpcbind     
udp6       0      0 :::761                  :::*                                611/rpcbind     

Comparing to the ports in the database:

"http://172.24.102.157:43722"
"http://172.24.102.157:59401"
"http://172.24.102.157:51794"
"http://172.24.102.157:50474"

So they are completely different. What's also interesting is that, watch monit summary does consistently show running then Does not exist.

The Monit daemon 5.2.4 uptime: 21h 58m

Process 'mongodb_gateway'           running
Process 'rabbit_gateway'            Does not exist
Process 'postgresql_gateway'        running
Process 'redis_gateway'             running
System 'system_8d482271-7d3c-4b4c-9651-65ce5d63f6d5' running

The question is, if the Jobs are failing? That would explain, why the ports are changing that often in the database.

I will keep debugging.

[jhiemer]

Ok, started the mongodb_gateway by hand and I could see the ruby process being started:

root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/sys/log/mongodb_gateway# top | grep ruby
20046 root      20   0   89612  27332   7968 R 19.3  2.7   0:00.58 ruby                                                                                                                        
20046 root      20   0  222160  57368  11104 S 71.1  5.6   0:02.72 ruby                                                                                                                        
20046 root      20   0  223672  59204  11388 S  8.6  5.8   0:02.98 ruby                                                                                                                        
20046 root      20   0  223800  59248  11388 S  0.3  5.8   0:02.99 ruby                                                                                                                        
20046 root      20   0  223928  59288  11388 S  0.3  5.8   0:03.00 ruby                                                                                                                        
20046 root      20   0  223928  59288  11388 S  0.3  5.8   0:03.01 ruby                                                                                                                        
20046 root      20   0  223928  59288  11388 S  0.3  5.8   0:03.02 ruby  

Running for a few seconds, and then they are gone.

[jhiemer]

And completely manually starting results in the same behaviour:

root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/packages/mongodb_gateway/services/ng/mongodb/bin# /var/vcap/packages/mongodb_gateway/services/ng/mongodb/bin/mongodb_gateway -c /var/vcap/jobs/mongodb_gateway/config/mongodb_gateway.yml 
Resque::Helpers will be gone with no replacement in Resque 2.0.0.
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:13: warning: already initialized constant HTTP_BAD_REQUEST
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:14: warning: already initialized constant HTTP_NOT_AUTHORIZED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:15: warning: already initialized constant HTTP_FORBIDDEN
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:16: warning: already initialized constant HTTP_NOT_FOUND
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:17: warning: already initialized constant HTTP_INTERNAL
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:18: warning: already initialized constant HTTP_NOT_IMPLEMENTED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:19: warning: already initialized constant HTTP_SERVICE_UNAVAIL
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:20: warning: already initialized constant HTTP_GATEWAY_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:29: warning: already initialized constant INVALID_CONTENT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:30: warning: already initialized constant MALFORMATTED_REQ
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:31: warning: already initialized constant UNKNOWN_PLAN_UNIQUE_ID
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:32: warning: already initialized constant UNKNOWN_PLAN
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:33: warning: already initialized constant UNSUPPORTED_VERSION
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:36: warning: already initialized constant NOT_AUTHORIZED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:41: warning: already initialized constant NOT_FOUND
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:44: warning: already initialized constant INTERNAL_ERROR
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:45: warning: already initialized constant EXTENSION_NOT_IMPL
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:46: warning: already initialized constant NODE_OPERATION_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:47: warning: already initialized constant SERVICE_START_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:48: warning: already initialized constant WARDEN_RUN_COMMAND_FAILURE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:51: warning: already initialized constant SERVICE_UNAVAILABLE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:54: warning: already initialized constant GATEWAY_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:57: warning: already initialized constant OVER_QUOTA
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:58: warning: already initialized constant JOB_QUEUE_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:59: warning: already initialized constant JOB_TIMEOUT
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:60: warning: already initialized constant BAD_SERIALIZED_DATAFILE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:61: warning: already initialized constant FILESIZE_TOO_LARGE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:62: warning: already initialized constant TOO_MANY_REDIRECTS
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:63: warning: already initialized constant FILE_CORRUPTED
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:64: warning: already initialized constant REDIS_CONCURRENT_UPDATE
/var/vcap/data/packages/mongodb_gateway/9101c714d750ea13d9a4557c32aa0fe63de7b2d5.1-1ac17c89c9aa6194b19a10728dc6408ab47fd92a/services/ng/mongodb/vendor/bundle/ruby/1.9.1/gems/vcap_services_base-0.2.5/lib/base/service_error.rb:65: warning: already initialized constant INVALID_SNAPSHOT_NAME
Thin web server (v1.6.1 codename Death Proof)
Maximum connections set to 1024
Listening on 172.24.102.157:33414, CTRL+C to stop
root@8d482271-7d3c-4b4c-9651-65ce5d63f6d5:/var/vcap/packages/mongodb_gateway/services/ng/mongodb/bin# 

Now I need to think about the next steps. :-)

[jhiemer]

So I can still see the service brokers stopping continuously, while the services itself a running without any problems. Is there any crash log, where I can see, why the service stopped working?

[poblin-orange]

mmm. We also met some instability with this release. No community effort on cf-services-contrib anyway.
We had good success with https://github.com/cf-platform-eng/cf-containers-broker, as an alternative, provisionning mongo database as docker container.

[drnic]

Yeah the docker-boshrelease or its spinoffs like postgresql-docker-boshrelease and logstash-docker-boshrelease have been my go-to for some quick primitive services.

On Thu, Sep 10, 2015 at 11:25 AM, Pierre Oblin [email protected]
wrote:

mmm. We also met some instability with this release. No community effort on cf-services-contrib anyway.

We had good success with https://github.com/cf-platform-eng/cf-containers-broker, as an alternative, provisionning mongo database as docker container.

Reply to this email directly or view it on GitHub:
#156 (comment)