[elasticsearch][apiman_gateway] error about apiman

[EricWittmann]

That is very mysterious. The error indicates that there was a failure while trying to create the elasticsearch index. The apiman quickstart comes with an embedded version of elasticsearch that runs on port 19200. I have no idea why the index creation would fail! Could you try it again with a fresh reinstall?

Unfortunately the root cause as reported by elasticsearch is not being included in the error. I'll fix that for future versions but it won't help us here. I could provide you a newer version with some additional logging enabled if a fresh installation still fails.

Another thing we could do is to probe the elasticsearch instance after you startup wildfly and see what its health is.

After you start up, try this:

curl http://localhost:19200

Post the results here, and then if that works OK, try this:

curl http://localhost:19200/apiman_gateway?pretty=true

[maslick]

maslick@maslick:~$ curl http://localhost:19200
{
  "status" : 200,
  "name" : "Lloigoroth",
  "cluster_name" : "apiman",
  "version" : {
    "number" : "1.7.1",
    "build_hash" : "b88f43fc40b0bcd7f173a1f9ee2e97816de80b19",
    "build_timestamp" : "2015-07-29T09:54:16Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

maslick@maslick:~$ curl http://localhost:19200/apiman_gateway?pretty=true
{
  "apiman_gateway" : {
    "aliases" : { },
    "mappings" : {
      "service" : {
        "properties" : {
          "endpoint" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "endpointType" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "organizationId" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "publicService" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "serviceId" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "version" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "application" : {
        "properties" : {
          "applicationId" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "contracts" : {
            "properties" : {
              "apiKey" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "serviceId" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "serviceOrgId" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "serviceVersion" : {
                "type" : "string",
                "index" : "not_analyzed"
              }
            }
          },
          "organizationId" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "version" : {
            "type" : "string",
            "index" : "not_analyzed"
          }
        }
      },
      "serviceContract" : {
        "properties" : {
          "apiKey" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "application" : {
            "properties" : {
              "applicationId" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "organizationId" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "version" : {
                "type" : "string",
                "index" : "not_analyzed"
              }
            }
          },
          "plan" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "service" : {
            "properties" : {
              "endpoint" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "endpointType" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "organizationId" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "publicService" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "serviceId" : {
                "type" : "string",
                "index" : "not_analyzed"
              },
              "version" : {
                "type" : "string",
                "index" : "not_analyzed"
              }
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1448882342720",
        "uuid" : "xa8tGUSYRAS_7UQBx5X0Iw",
        "number_of_replicas" : "1",
        "number_of_shards" : "5",
        "version" : {
          "created" : "1070199"
        }
      }
    },
    "warmers" : { }
  }
}

[EricWittmann]

Both of those look good! Can you do a basic test of apiman by creating a public service, publishing it to the gateway, and then seeing if you can invoke it? I'd be interested to know if that works despite the error.

It seems as though everything is set up properly, but that when apiman talks to elasticsearch it's getting error responses back when it shouldn't.

I'm grasping at straws a bit, I confess. :(

[maslick]

The thing is I cannot start the UI. I log-in at https://localhost/apimanui using admin/admin123! and I get Forbidden:

https://localhost/apimanui/api-manager?code=08AwXBB1fd6iCuzAEgJijh0Ky_zzt2g3BypozZ4Mnlk.69d9a377-9811-4b02-bf57-cccb4c7b8f5a&state=2%2F83eb39b3-2bb7-4def-9cf7-5f9e6d64177e

Backend:

apiman_1 | 15:09:48,105 ERROR [org.keycloak.adapters.OAuthRequestAuthenticator] (default task-33) failed to turn code into token: java.net.ConnectException: Connection refused
apiman_1 |  at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_85]
apiman_1 |  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_85]
apiman_1 |  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_85]
apiman_1 |  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_85]
apiman_1 |  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_85]
apiman_1 |  at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_85]
apiman_1 |  at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:637) [jsse.jar:1.7.0_85]
apiman_1 |  at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:524)
apiman_1 |  at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:403)
apiman_1 |  at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
apiman_1 |  at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
apiman_1 |  at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
apiman_1 |  at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
apiman_1 |  at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
apiman_1 |  at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
apiman_1 |  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
apiman_1 |  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
apiman_1 |  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
apiman_1 |  at org.keycloak.adapters.ServerRequest.invokeAccessCodeToToken(ServerRequest.java:122) [keycloak-adapter-core-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at org.keycloak.adapters.ServerRequest.invokeAccessCodeToToken(ServerRequest.java:95) [keycloak-adapter-core-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at org.keycloak.adapters.OAuthRequestAuthenticator.resolveCode(OAuthRequestAuthenticator.java:296) [keycloak-adapter-core-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at org.keycloak.adapters.OAuthRequestAuthenticator.authenticate(OAuthRequestAuthenticator.java:244) [keycloak-adapter-core-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at org.keycloak.adapters.RequestAuthenticator.authenticate(RequestAuthenticator.java:90) [keycloak-adapter-core-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at org.keycloak.adapters.undertow.AbstractUndertowKeycloakAuthMech.keycloakAuthenticate(AbstractUndertowKeycloakAuthMech.java:114) [keycloak-undertow-adapter-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at org.keycloak.adapters.undertow.ServletKeycloakAuthMech.authenticate(ServletKeycloakAuthMech.java:94) [keycloak-undertow-adapter-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:281) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:298) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:268) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:131) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:106) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:99) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:54) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:63) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
apiman_1 |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at org.keycloak.adapters.undertow.ServletPreAuthActionsHandler.handleRequest(ServletPreAuthActionsHandler.java:69) [keycloak-undertow-adapter-1.2.0.Final.jar:1.2.0.Final]
apiman_1 |  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]
apiman_1 |  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_85]
apiman_1 |  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_85]
apiman_1 |  at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_85]
apiman_1 | 

[EricWittmann]

Wow this is a keycloak error - when it's trying to consume the redirect code and fetch the oauth access token (I guess). Connection refused indicates that the keycloak authentication adapter (which protects the apiman UI) couldn't connect to the keycloak server (which in the case of the quickstart is just running in the same wildfly instance).

What is your local setup? I noticed that our URL is https://localhost rather than https://localhost:8443. Is that because you modified the wildfly config to run on 443 instead of 8443? Any other configuration changes? Are you running anything else on your local machine?

I can't reproduce any of this with the standard quickstart instructions.

[maslick]

Actually I am running apiman in a docker container. So apiman's port 8443 is mapped to the host's 443 port. So nothing criminal here.

Can it be that due to this elastic search error the Keycloak adapter did not (if at all) initialized properly?

[EricWittmann]

Ah docker, I see. Did you create your own image or are you using the
official apiman image? Can you post the docker commands you are using to
install the image and start it up so I can try to reproduce?

I can't see how the elasticsearch problem could affect keycloak in any
way. More likely both failures are symptoms of something else.

On Mon, Nov 30, 2015 at 11:06 AM, Pavel Maslov [email protected]
wrote:

Actually I am running a docker image of apiman. So apiman's 8443 is mapped
to the host 443 port. So nothing criminal here.

Can it be that due to this elastic search error Keycloak adapter did not
(if at all) initialized properly?

—
Reply to this email directly or view it on GitHub
#287 (comment).

[maslick]

https://github.com/maslick/apiman-docker

I have it up and running on a remote host (AWS cloud). But it fails to run on localhost.

[EricWittmann]

OK that's interesting. The dockerfile looks reasonable to me, although I'm not a docker expert by any means. I can't explain the elasticsearch problem, but I'm wondering if the keycloak error is possibly related to the port mappings. I seem to recall having an issue in the past related to this. You might try starting up WildFly on port 443 and adjust your docker port mappings accordingly. You should be able to set this property:

jboss.https.port

Or else you can modify the standalone-apiman.xml file in the appropriate place (search the file for 8443).

Regarding elasticsearch - if using docker I would recommend starting up an elasticsearch instance via docker and then pointing apiman to that. The embedded apiman elasticsearch instance is only appropriate for evaluating the quickstart.

Once you have ES running standalone, you can point apiman to it by configuring the settings in apiman.properties here:

https://github.com/apiman/apiman/blob/master/distro/wildfly8/src/main/resources/overlay/standalone/configuration/apiman.properties#L13-L18

[maslick]

I have managed to run the UI with the host port set to 8443 in docker-compose.yml. Still the same error:

apiman_1 | 17:32:48,459 INFO  [org.elasticsearch.http] (MSC service thread 1-2) [Boomslang] bound_address {inet[/0.0.0.0:19200]}, publish_address {inet[/172.17.0.3:19200]}
apiman_1 | 17:32:48,459 INFO  [org.elasticsearch.node] (MSC service thread 1-2) [Boomslang] started
apiman_1 | 17:32:48,459 INFO  [stdout] (MSC service thread 1-2) -----------------------------
apiman_1 | 17:32:48,459 INFO  [stdout] (MSC service thread 1-2) apiman-es started!
apiman_1 | 17:32:48,459 INFO  [stdout] (MSC service thread 1-2) -----------------------------
apiman_1 | 17:32:48,462 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) JBAS017534: Registered web context: /apiman-es
apiman_1 | 17:32:48,494 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apimanui.war" (runtime-name : "apimanui.war")
apiman_1 | 17:32:48,494 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 51) JBAS018559: Deployed "main-auth-server.war" (runtime-name : "main-auth-server.war")
apiman_1 | 17:32:48,495 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-gateway-api.war" (runtime-name : "apiman-gateway-api.war")
apiman_1 | 17:32:48,495 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-gateway.war" (runtime-name : "apiman-gateway.war")
apiman_1 | 17:32:48,495 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-es.war" (runtime-name : "apiman-es.war")
apiman_1 | 17:32:48,495 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman.war" (runtime-name : "apiman.war")
apiman_1 | 17:32:48,512 INFO  [org.elasticsearch.gateway] (elasticsearch[Boomslang][clusterService#updateTask][T#1]) [Boomslang] recovered [1] indices into cluster_state
apiman_1 | 17:32:48,542 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://0.0.0.0:9990/management
apiman_1 | 17:32:48,542 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://0.0.0.0:9990
apiman_1 | 17:32:48,543 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.2.0.Final "Tweek" started in 11482ms - Started 722 of 815 services (158 services are lazy, passive or on-demand)


apiman_1 | 17:32:57,445 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using multi thread/connection supporting pooling connection manager
apiman_1 | 17:32:57,445 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using multi thread/connection supporting pooling connection manager
apiman_1 | 17:32:57,473 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using default GSON instance
apiman_1 | 17:32:57,474 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Node Discovery disabled...
apiman_1 | 17:32:57,474 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Idle connection reaping disabled...
apiman_1 | 17:32:57,476 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using default GSON instance
apiman_1 | 17:32:57,476 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Node Discovery disabled...
apiman_1 | 17:32:57,476 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Idle connection reaping disabled...
apiman_1 | 17:32:57,687 INFO  [org.elasticsearch.cluster.metadata] (elasticsearch[Boomslang][clusterService#updateTask][T#1]) [Boomslang] [apiman_gateway] creating index, cause [api], templates [], shards [5]/[1], mappings [application, service, serviceContract]
apiman_1 | 17:32:57,785 ERROR [stderr] (PollCachingESInvalidator) Exception in thread "PollCachingESInvalidator" java.lang.RuntimeException: java.lang.Exception: Failed to create index: apiman_gateway
apiman_1 | 17:32:57,785 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.ESClientFactory.initializeClient(ESClientFactory.java:200)
apiman_1 | 17:32:57,786 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.ESClientFactory.createJestClient(ESClientFactory.java:140)
apiman_1 | 17:32:57,786 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.ESClientFactory.createJestClient(ESClientFactory.java:101)
apiman_1 | 17:32:57,786 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.ESClientFactory.createClient(ESClientFactory.java:66)
apiman_1 | 17:32:57,786 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.AbstractESComponent.getClient(AbstractESComponent.java:45)
apiman_1 | 17:32:57,786 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.PollCachingESRegistry.checkCacheVersion(PollCachingESRegistry.java:195)
apiman_1 | 17:32:57,787 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.PollCachingESRegistry$6.run(PollCachingESRegistry.java:168)
apiman_1 | 17:32:57,787 ERROR [stderr] (PollCachingESInvalidator)   at java.lang.Thread.run(Thread.java:745)
apiman_1 | 17:32:57,787 ERROR [stderr] (PollCachingESInvalidator) Caused by: java.lang.Exception: Failed to create index: apiman_gateway
apiman_1 | 17:32:57,787 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.ESClientFactory.createIndex(ESClientFactory.java:216)
apiman_1 | 17:32:57,788 ERROR [stderr] (PollCachingESInvalidator)   at io.apiman.gateway.engine.es.ESClientFactory.initializeClient(ESClientFactory.java:197)
apiman_1 | 17:32:57,788 ERROR [stderr] (PollCachingESInvalidator)   ... 7 more
apiman_1 | 17:33:33,880 INFO  [stdout] (default task-108) 2015-11-30 17:33:33,880 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
apiman_1 | 17:33:33,930 INFO  [org.hibernate.jpa.internal.util.LogHelper] (default task-108) HHH000204: Processing PersistenceUnitInfo [
apiman_1 |  name: apiman-manager-api-jpa
apiman_1 |  ...]
apiman_1 | 17:33:34,099 INFO  [org.hibernate.dialect.Dialect] (default task-108) HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
apiman_1 | 17:33:34,101 INFO  [org.hibernate.engine.jdbc.internal.LobCreatorBuilder] (default task-108) HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
apiman_1 | 17:33:34,143 INFO  [org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory] (default task-108) HHH000397: Using ASTQueryTranslatorFactory
apiman_1 | 17:33:34,384 INFO  [org.hibernate.tool.hbm2ddl.SchemaUpdate] (default task-108) HHH000228: Running hbm2ddl schema update
apiman_1 | 17:33:34,384 INFO  [org.hibernate.tool.hbm2ddl.SchemaUpdate] (default task-108) HHH000102: Fetching database metadata
apiman_1 | 17:33:34,386 INFO  [org.hibernate.tool.hbm2ddl.SchemaUpdate] (default task-108) HHH000396: Updating schema
apiman_1 | 17:33:34,390 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: application_versions
apiman_1 | 17:33:34,392 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: applications
apiman_1 | 17:33:34,394 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: auditlog
apiman_1 | 17:33:34,396 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: contracts
apiman_1 | 17:33:34,398 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: endpoint_properties
apiman_1 | 17:33:34,399 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: gateways
apiman_1 | 17:33:34,401 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: memberships
apiman_1 | 17:33:34,403 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: organizations
apiman_1 | 17:33:34,404 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: pd_templates
apiman_1 | 17:33:34,406 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: permissions
apiman_1 | 17:33:34,408 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plan_versions
apiman_1 | 17:33:34,410 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plans
apiman_1 | 17:33:34,412 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plugins
apiman_1 | 17:33:34,413 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: policies
apiman_1 | 17:33:34,415 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: policydefs
apiman_1 | 17:33:34,416 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: roles
apiman_1 | 17:33:34,417 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: service_defs
apiman_1 | 17:33:34,419 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: service_versions
apiman_1 | 17:33:34,420 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: services
apiman_1 | 17:33:34,422 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: svc_gateways
apiman_1 | 17:33:34,423 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: svc_plans
apiman_1 | 17:33:34,424 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: users
apiman_1 | 17:33:34,426 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: application_versions
apiman_1 | 17:33:34,429 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: applications
apiman_1 | 17:33:34,430 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: auditlog
apiman_1 | 17:33:34,432 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: contracts
apiman_1 | 17:33:34,433 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: endpoint_properties
apiman_1 | 17:33:34,435 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: gateways
apiman_1 | 17:33:34,436 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: memberships
apiman_1 | 17:33:34,437 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: organizations
apiman_1 | 17:33:34,439 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: pd_templates
apiman_1 | 17:33:34,440 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: permissions
apiman_1 | 17:33:34,441 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plan_versions
apiman_1 | 17:33:34,442 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plans
apiman_1 | 17:33:34,443 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plugins
apiman_1 | 17:33:34,445 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: policies
apiman_1 | 17:33:34,446 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: policydefs
apiman_1 | 17:33:34,447 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: roles
apiman_1 | 17:33:34,449 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: service_defs
apiman_1 | 17:33:34,450 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: service_versions
apiman_1 | 17:33:34,451 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: services
apiman_1 | 17:33:34,452 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: svc_gateways
apiman_1 | 17:33:34,453 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: svc_plans
apiman_1 | 17:33:34,454 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: users
apiman_1 | 17:33:34,456 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: application_versions
apiman_1 | 17:33:34,457 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: applications
apiman_1 | 17:33:34,458 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: auditlog
apiman_1 | 17:33:34,460 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: contracts
apiman_1 | 17:33:34,461 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: endpoint_properties
apiman_1 | 17:33:34,462 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: gateways
apiman_1 | 17:33:34,463 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: memberships
apiman_1 | 17:33:34,464 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: organizations
apiman_1 | 17:33:34,466 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: pd_templates
apiman_1 | 17:33:34,467 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: permissions
apiman_1 | 17:33:34,469 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plan_versions
apiman_1 | 17:33:34,470 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plans
apiman_1 | 17:33:34,471 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: plugins
apiman_1 | 17:33:34,473 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: policies
apiman_1 | 17:33:34,475 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: policydefs
apiman_1 | 17:33:34,476 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: roles
apiman_1 | 17:33:34,477 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: service_defs
apiman_1 | 17:33:34,478 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: service_versions
apiman_1 | 17:33:34,479 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: services
apiman_1 | 17:33:34,480 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: svc_gateways
apiman_1 | 17:33:34,482 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: svc_plans
apiman_1 | 17:33:34,483 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: users
apiman_1 | 17:33:34,486 INFO  [java.sql.DatabaseMetaData] (default task-108) HHH000262: Table not found: hibernate_sequence
db_1     | ERROR:  constraint "uk_7h67tqdhdejapl8h2xtmscrqu" of relation "application_versions" does not exist
db_1     | STATEMENT:  
db_1     |      alter table application_versions 
db_1     |          drop constraint UK_7h67tqdhdejapl8h2xtmscrqu
db_1     | ERROR:  constraint "uk_i3bq88cawo5mi1kf3pk8fkvsm" of relation "contracts" does not exist
db_1     | STATEMENT:  
db_1     |      alter table contracts 
db_1     |          drop constraint UK_i3bq88cawo5mi1kf3pk8fkvsm
db_1     | ERROR:  constraint "uk_fwy6c61bcnvshcsxubanjvi08" of relation "memberships" does not exist
db_1     | STATEMENT:  
db_1     |      alter table memberships 
db_1     |          drop constraint UK_fwy6c61bcnvshcsxubanjvi08
db_1     | ERROR:  constraint "uk_53ss55b2hiye2e9v6lftdo5jg" of relation "plan_versions" does not exist
db_1     | STATEMENT:  
db_1     |      alter table plan_versions 
db_1     |          drop constraint UK_53ss55b2hiye2e9v6lftdo5jg
db_1     | ERROR:  constraint "uk_ofbok9ushig9vviq01dnu11x" of relation "plugins" does not exist
db_1     | STATEMENT:  
db_1     |      alter table plugins 
db_1     |          drop constraint UK_ofbok9ushig9vviq01dnu11x
db_1     | ERROR:  constraint "uk_mypgo5c13ghdsno93sps5xxok" of relation "service_versions" does not exist
db_1     | STATEMENT:  
db_1     |      alter table service_versions 
db_1     |          drop constraint UK_mypgo5c13ghdsno93sps5xxok
apiman_1 | 17:33:35,208 INFO  [org.hibernate.tool.hbm2ddl.SchemaUpdate] (default task-108) HHH000232: Schema update complete
apiman_1 | 17:33:35,226 INFO  [stdout] (default task-108) Getting info for user admin

At least I can see the UI.

[EricWittmann]

Progress!

Regarding the ES error - seem my previous comment about using an external ES instance instead of the embedded one that comes with apiman. That is definitely the preferred approach.

Also regarding your DB errors - you may want to initialize postgres with the apiman DDL rather than let hibernate update the DB. I wouldn't really trust hibernate.

[maslick]

I just checked the logs of my EC2 docker instance. It has the same error, although
everything seems to be working (I didn't explicitly check Elasticsearch though). I have a private service and application with a subscription to this service. I am able to call the service through apiman and I can see the statistics.

[EricWittmann]

OK that's actually not that surprising, given the curl results. Still - I can't explain it. I definitely still think an external ES is the way to go! :)

[maslick]

The question is how can you import the DDL into postgres in the Docker environment?

[maslick]

Beside putting Elasticsearch in another container and changing https://github.com/apiman/apiman/blob/master/distro/wildfly8/src/main/resources/overlay/standalone/configuration/apiman.properties#L13-L18 is there any other settings I need to change?

[EricWittmann]

First, for postgres+DDL I think you can extend the postgres docker image and include your own custom DDL in the docker-entrypoint-initdb.d directory.

Read here under "How to extend this image" or search for docker-entrypoint-initdb.d:

https://hub.docker.com/_/postgres/

Second, I don't think you need any other changes besides starting up ES in its own container and then updating apiman.properties.

[maslick]

quick question: does apiman.properties support env. variables (global)?

[EricWittmann]

Yes, here is the syntax: ``` apiman.es.host=${env:ES_HOST} ``` Obviously assuming that ES_HOST is your env variable.

[maslick]

Running with elasticsearch in a separate container (everything works, but got plenty of error messages):

apiman_1  | 19:50:50,653 INFO  [org.elasticsearch.http] (MSC service thread 1-1) [D-Man] bound_address {inet[/0.0.0.0:19200]}, publish_address {inet[/172.17.0.4:19200]}
apiman_1  | 19:50:50,653 INFO  [org.elasticsearch.node] (MSC service thread 1-1) [D-Man] started
apiman_1  | 19:50:50,654 INFO  [stdout] (MSC service thread 1-1) -----------------------------
apiman_1  | 19:50:50,654 INFO  [stdout] (MSC service thread 1-1) apiman-es started!
apiman_1  | 19:50:50,654 INFO  [stdout] (MSC service thread 1-1) -----------------------------
apiman_1  | 19:50:50,658 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017534: Registered web context: /apiman-es
apiman_1  | 19:50:50,688 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apimanui.war" (runtime-name : "apimanui.war")
apiman_1  | 19:50:50,688 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 51) JBAS018559: Deployed "main-auth-server.war" (runtime-name : "main-auth-server.war")
apiman_1  | 19:50:50,689 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-gateway-api.war" (runtime-name : "apiman-gateway-api.war")
apiman_1  | 19:50:50,689 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-gateway.war" (runtime-name : "apiman-gateway.war")
apiman_1  | 19:50:50,689 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman-es.war" (runtime-name : "apiman-es.war")
apiman_1  | 19:50:50,689 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 29) JBAS018559: Deployed "apiman.war" (runtime-name : "apiman.war")
apiman_1  | 19:50:50,728 INFO  [org.elasticsearch.gateway] (elasticsearch[D-Man][clusterService#updateTask][T#1]) [D-Man] recovered [1] indices into cluster_state
apiman_1  | 19:50:50,746 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://0.0.0.0:9990/management
apiman_1  | 19:50:50,746 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://0.0.0.0:9990
apiman_1  | 19:50:50,747 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.2.0.Final "Tweek" started in 11485ms - Started 722 of 815 services (158 services are lazy, passive or on-demand)




apiman_1  | 19:50:59,797 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using multi thread/connection supporting pooling connection manager
apiman_1  | 19:50:59,797 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using multi thread/connection supporting pooling connection manager
apiman_1  | 19:50:59,829 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using default GSON instance
apiman_1  | 19:50:59,829 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Using default GSON instance
apiman_1  | 19:50:59,830 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Node Discovery disabled...
apiman_1  | 19:50:59,830 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Node Discovery disabled...
apiman_1  | 19:50:59,830 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Idle connection reaping disabled...
apiman_1  | 19:50:59,830 INFO  [io.searchbox.client.JestClientFactory] (PollCachingESInvalidator) Idle connection reaping disabled...
elastic_1 | [2015-11-30 19:51:00,163][INFO ][cluster.metadata         ] [Sara Grey] [apiman_gateway] creating index, cause [api], templates [], shards [5]/[1], mappings [application, service, serviceContract]
elastic_1 | [2015-11-30 19:51:00,305][INFO ][rest.suppressed          ] /apiman_gateway Params: {index=apiman_gateway}
elastic_1 | [apiman_gateway] IndexAlreadyExistsException[already exists]
elastic_1 |     at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:168)
elastic_1 |     at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:520)
elastic_1 |     at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.access$200(MetaDataCreateIndexService.java:97)
elastic_1 |     at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$2.execute(MetaDataCreateIndexService.java:241)
elastic_1 |     at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:388)
elastic_1 |     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
elastic_1 |     at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
elastic_1 |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elastic_1 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elastic_1 |     at java.lang.Thread.run(Thread.java:745)
apiman_1  | 19:51:00,312 ERROR [stderr] (PollCachingESInvalidator) Exception in thread "PollCachingESInvalidator" java.lang.RuntimeException: java.lang.UnsupportedOperationException: JsonObject
apiman_1  | 19:51:00,313 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.ESClientFactory.initializeClient(ESClientFactory.java:200)
apiman_1  | 19:51:00,313 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.ESClientFactory.createJestClient(ESClientFactory.java:140)
apiman_1  | 19:51:00,313 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.ESClientFactory.createJestClient(ESClientFactory.java:101)
apiman_1  | 19:51:00,313 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.ESClientFactory.createClient(ESClientFactory.java:66)
elastic_1 | [2015-11-30 19:51:00,314][INFO ][rest.suppressed          ] /apiman_gateway/dataVersion/instance Params: {index=apiman_gateway, id=instance, type=dataVersion}
elastic_1 | NoShardAvailableActionException[No shard available for [get [apiman_gateway][dataVersion][instance]: routing [null]]]; nested: RemoteTransportException[[Sara Grey][172.17.0.3:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
elastic_1 |     at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:199)
elastic_1 |     at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:186)
elastic_1 |     at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.access$1400(TransportSingleShardAction.java:115)
elastic_1 |     at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:240)
elastic_1 |     at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:821)
elastic_1 |     at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:799)
elastic_1 |     at org.elasticsearch.transport.TransportService$4.onFailure(TransportService.java:361)
elastic_1 |     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:42)
elastic_1 |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elastic_1 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elastic_1 |     at java.lang.Thread.run(Thread.java:745)
elastic_1 | Caused by: RemoteTransportException[[Sara Grey][172.17.0.3:9300][indices:data/read/get[s]]]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]];
elastic_1 | Caused by: [apiman_gateway][[apiman_gateway][3]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED, RELOCATED]]
elastic_1 |     at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:974)
elastic_1 |     at org.elasticsearch.index.shard.IndexShard.get(IndexShard.java:605)
elastic_1 |     at org.elasticsearch.index.get.ShardGetService.innerGet(ShardGetService.java:173)
elastic_1 |     at org.elasticsearch.index.get.ShardGetService.get(ShardGetService.java:86)
elastic_1 |     at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:101)
elastic_1 |     at org.elasticsearch.action.get.TransportGetAction.shardOperation(TransportGetAction.java:44)
elastic_1 |     at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:282)
elastic_1 |     at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$ShardTransportHandler.messageReceived(TransportSingleShardAction.java:275)
elastic_1 |     at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)
elastic_1 |     at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
elastic_1 |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
elastic_1 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
elastic_1 |     at java.lang.Thread.run(Thread.java:745)
apiman_1  | 19:51:00,313 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.AbstractESComponent.getClient(AbstractESComponent.java:45)
apiman_1  | 19:51:00,313 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.PollCachingESRegistry.checkCacheVersion(PollCachingESRegistry.java:195)
apiman_1  | 19:51:00,314 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.PollCachingESRegistry$6.run(PollCachingESRegistry.java:168)
apiman_1  | 19:51:00,314 ERROR [stderr] (PollCachingESInvalidator)  at java.lang.Thread.run(Thread.java:745)
apiman_1  | 19:51:00,314 ERROR [stderr] (PollCachingESInvalidator) Caused by: java.lang.UnsupportedOperationException: JsonObject
apiman_1  | 19:51:00,314 ERROR [stderr] (PollCachingESInvalidator)  at com.google.gson.JsonElement.getAsString(JsonElement.java:191)
apiman_1  | 19:51:00,314 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.client.JestResult.setJsonObject(JestResult.java:95)
apiman_1  | 19:51:00,314 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:68)
apiman_1  | 19:51:00,315 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.action.GenericResultAbstractAction.createNewElasticSearchResult(GenericResultAbstractAction.java:20)
apiman_1  | 19:51:00,315 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:146)
apiman_1  | 19:51:00,315 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:65)
apiman_1  | 19:51:00,315 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.ESClientFactory.createIndex(ESClientFactory.java:214)
apiman_1  | 19:51:00,315 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.ESClientFactory.initializeClient(ESClientFactory.java:197)
apiman_1  | 19:51:00,315 ERROR [stderr] (PollCachingESInvalidator)  ... 7 more
apiman_1  | 19:51:00,316 ERROR [stderr] (PollCachingESInvalidator) Exception in thread "PollCachingESInvalidator" java.lang.UnsupportedOperationException: JsonObject
apiman_1  | 19:51:00,316 ERROR [stderr] (PollCachingESInvalidator)  at com.google.gson.JsonElement.getAsString(JsonElement.java:191)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.client.JestResult.setJsonObject(JestResult.java:95)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.action.AbstractAction.createNewElasticSearchResult(AbstractAction.java:68)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.action.GenericResultAbstractDocumentTargetedAction.createNewElasticSearchResult(GenericResultAbstractDocumentTargetedAction.java:17)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.client.http.JestHttpClient.deserializeResponse(JestHttpClient.java:146)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.searchbox.client.http.JestHttpClient.execute(JestHttpClient.java:65)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.PollCachingESRegistry.checkCacheVersion(PollCachingESRegistry.java:195)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at io.apiman.gateway.engine.es.PollCachingESRegistry$6.run(PollCachingESRegistry.java:168)
apiman_1  | 19:51:00,317 ERROR [stderr] (PollCachingESInvalidator)  at java.lang.Thread.run(Thread.java:745)
apiman_1  | 19:51:47,993 INFO  [stdout] (default task-99) 20

[EricWittmann]

OK I'm wondering if the embedded ES instance is trying to cluster with your
standalone one. To avoid that, modify your apiman docker file to remove
the following:

wildfly-8.2.0.Final/standalone/deployments/apiman-es.*

That will remove the embedded ES from wildfly.

I cannot explain the UnsupportedOperationException you're getting in the
PollCachingESInvalidator - that doesn't make any sense at all. My
suggestion for now is to switch to using the standard ESRegistry in
apiman.properties:

https://github.com/apiman/apiman/blob/master/distro/wildfly8/src/main/resources/overlay/standalone/configuration/apiman.properties#L126

Just drop the "PollCaching" from that line so that it becomes:

apiman-gateway.registry=io.apiman.gateway.engine.es.ESRegistry

[maslick]

Resolved. Thanks @EricWittmann !