Backwards compatilibity with 2.* ? about schema-registry-ui

[andmarios]

I just pushed the new docker image. Could you give it a spin? It should help you with CORS.

You should run it like this:

docker pull landoop/schema-registry-ui
docker run --rm -it -p 8000:8000 \
             -e "PROXY=true" \
             -e "SCHEMAREGISTRY_URL=http://schema.registry.url" \
             landoop/schema-registry-ui

[jramapuram]

Thanks for the quick turnaround! Works perfectly!

[Antwnis]

Sure will check what needs to be done for backward compatibility
I think you are currently facing a CORS (Cross Origin Issue). Can you enlighten us with how you are serving the ui app. Are you using the docker or nginx ?

[jramapuram]

I made sure to add the access lines specified to the schema registry properties file. Will give it a shot on the host itself to make sure there is no issue. Serving with docker.

[jramapuram]

Just tested on the same machine serving the content. Get the following in the UI:

URL : http://localhost:8081 
CONNECTIVITY ERROR

Here is schema-registry conf:

# cat /etc/schema-registry/schema-registry.properties
# Copyright 2014 Confluent Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

port=8081
kafkastore.connection.url=localhost:2181
kafkastore.topic=_schemas
debug=false
access.control.allow.methods=GET,POST,OPTIONS
access.control.allow.origin=*

[andmarios]

Could you please check your browser's console about errors? For example in Chrome you could either right-click and select inspect or press Ctrl+Shift+i. On the dev window that opens, check the console. The error messages there may help us understand the issue.

We haven't tested with Confluent 2.0.1 yet, but I think the schema-registry API did not change between 2.0.1 and 3.0.0. I will test it later and report back.

[jramapuram]

Hmm looks like a connection error:

Starting schema-registry controller : config 
angular.js:13920  curl -X GET http://localhost:8081/config
angular.js:12011GET http://localhost:8081/config net::ERR_CONNECTION_REFUSED(anonymous function) @ angular.js:12011n @ angular.js:11776(anonymous function) @ angular.js:11571(anonymous function) @ angular.js:16383$eval @ angular.js:17682$digest @ angular.js:17495$apply @ angular.js:17790(anonymous function) @ angular.js:1761invoke @ angular.js:4718c @ angular.js:1759Bc @ angular.js:1779fe @ angular.js:1664(anonymous function) @ angular.js:31763b @ angular.js:3207Sf @ angular.js:3497d @ angular.js:3485
angular.js:13920Failure with : "Get global config rejection : null -1"(anonymous function) @ angular.js:13920failure @ combined.js:1414(anonymous function) @ angular.js:16383$eval @ angular.js:17682$digest @ angular.js:17495$apply @ angular.js:17790l @ angular.js:11831J @ angular.js:12033e @ angular.js:11976
angular.js:13920Starting schema-registry controller : list ( initializing subject cache )
angular.js:13920  curl -X GET http://localhost:8081/subjects/
angular.js:13920Starting schema-registry controller - home
angular.js:12011 GET http://localhost:8081/subjects/ net::ERR_CONNECTION_REFUSED(anonymous function) @ angular.js:12011n @ angular.js:11776(anonymous function) @ angular.js:11571(anonymous function) @ angular.js:16383$eval @ angular.js:17682$digest @ angular.js:17495$apply @ angular.js:17790l @ angular.js:11831J @ angular.js:12033t.onload @ angular.js:11966

However, on the host itself it works fine:

$ curl -X GET http://localhost:8081/config
{"compatibilityLevel":"BACKWARD"}

[andmarios]

I just gave it a try with Confluent 2.0.1 and it works. I temporarily opened the port if you wanna check it out:

docker run --rm -it \
             -e "SCHEMAREGISTRY_URL=http://schema-registry-201.demo.landoop.com:28081" \
             -p 8000:8000 landoop/schema-registry-ui

Our Schema Registry settings are below. We have a custom znode (localhost:22181/confluent), which you can ignore.

port=28081
kafkastore.connection.url=localhost:22181/confluent
kafkastore.topic=_schemas
debug=false
access.control.allow.methods=GET,POST,PUT,DELETE,OPTIONS
access.control.allow.origin=*

[jramapuram]

I think the issue earlier was obvious (in hindsight): running in a docker container you can't bind to localhost as that is the container ip. I switched it back to the original, but looks like I have a CORS issue now:

Starting schema-registry controller : config 
angular.js:13920  curl -X GET http://<MYACTUALHOST>:8081/config
angular.js:13920 Starting schema-registry controller : list ( initializing subject cache )
angular.js:13920   curl -X GET http://<MYACTUALHOST>:8081/subjects/
angular.js:13920 Starting schema-registry controller - home
:8000/#/:1 XMLHttpRequest cannot load http://<MYACTUALHOST>:8081/config. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://<MYACTUALHOST>:8000' is therefore not allowed access.
angular.js:13920 Failure with : "Get global config rejection : null -1"(anonymous function) @ angular.js:13920failure @ combined.js:1414(anonymous function) @ angular.js:16383$eval @ angular.js:17682$digest @ angular.js:17495$apply @ angular.js:17790l @ angular.js:11831J @ angular.js:12033e @ angular.js:11976
:8000/#/:1 XMLHttpRequest cannot load http://<MYACTUALHOST>:8081/subjects/. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://<MYACTUALHOST>:8000' is therefore not allowed access.

I set the schema registry allow calls as you specified above and restarted the service. Any thoughts on what else I might need to do?

[andmarios]

Sorry, I can't think of anything. The way we test for CORS when we are not sure at landoop, is via curl. The origin hostname can be anything:

curl -v -H "Origin: https://landoop.com" \
     -H "Access-Control-Request-Method: POST" \
     -H "Access-Control-Request-Headers: X-Requested-With" \
     -X OPTIONS -I \
     http://<HOSTNAME>:8081/

If your schema registry returns proper CORS headers, you should see entries in the response like below.

...
Access-Control-Max-Age: 1800
Access-Control-Allow-Methods: GET,POST,HEAD
Access-Control-Allow-Headers: X-Requested-With,Content-Type,Accept,Origin
....

[jramapuram]

It seems like it is allowing POST,GET,OPTIONS & HEAD ¯\ (ツ)/¯
However, I dont see the Access-Control-Allow-Headers you mention.

I see the following when I run the curl above:

X-Requested-With" -X OPTIONS -I http://<MYHOSTNAME>:8081/
* Hostname was NOT found in DNS cache
*   Trying <MYHOSTIP>...
* Connected to <MYHOSTNAME> (<MYHOSTIP>) port 8081 (#0)
> OPTIONS / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: <MYHOSTNAME>:8081
> Accept: */*
> Origin: https://landoop.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Length: 1116
Content-Length: 1116
< Content-Type: application/vnd.sun.wadl+xml
Content-Type: application/vnd.sun.wadl+xml
< Allow: POST,GET,OPTIONS,HEAD
Allow: POST,GET,OPTIONS,HEAD
< Last-Modified: Mon, 12 Sep 2016 20:26:07 EEST
Last-Modified: Mon, 12 Sep 2016 20:26:07 EEST
* Server Jetty(8.1.16.v20140903) is not blacklisted
< Server: Jetty(8.1.16.v20140903)
Server: Jetty(8.1.16.v20140903)

<
* Excess found in a non pipelined read: excess = 1116 url = / (zero-length body)
* Connection #0 to host <MYHOSTNAME> left intact

This is what I see I run the service (it looks like it's returning a 200 when the client requests the corresponding REST endpoints [/subjects & /config] ) :

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2016-09-12 20:20:15,226] INFO SchemaRegistryConfig values:
    master.eligibility = true
    port = 8081
    kafkastore.timeout.ms = 500
    kafkastore.init.timeout.ms = 60000
    debug = false
    kafkastore.zk.session.timeout.ms = 30000
    request.logger.name = io.confluent.rest-utils.requests
    metrics.sample.window.ms = 30000
    schema.registry.zk.namespace = schema_registry
    kafkastore.topic = _schemas
    avro.compatibility.level = backward
    shutdown.graceful.ms = 1000
    response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
    metrics.jmx.prefix = kafka.schema.registry
    host.name = <MYHOST>
    metric.reporters = []
    kafkastore.commit.interval.ms = -1
    kafkastore.connection.url = localhost:2181
    metrics.num.samples = 2
    response.mediatype.default = application/vnd.schemaregistry.v1+json
    kafkastore.topic.replication.factor = 3
 (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
[2016-09-12 20:20:16,094] INFO Initialized the consumer offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:87)
[2016-09-12 20:20:17,229] WARN The replication factor of the schema topic _schemas is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore:201)
[2016-09-12 20:20:17,315] INFO [kafka-store-reader-thread-_schemas], Starting  (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:68)
[2016-09-12 20:20:17,508] INFO Wait to catch up until the offset of the last message at 60 (io.confluent.kafka.schemaregistry.storage.KafkaStore:221)
[2016-09-12 20:20:18,339] INFO Created schema registry namespace localhost:2181/schema_registry (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:194)
[2016-09-12 20:20:18,422] INFO Successfully elected the new master: {"host":"<MYHOST>","port":8081,"master_eligibility":true,"version":1} (io.confluent.kafka.schemaregistry.zookeeper.ZookeeperMasterElector:80)
[2016-09-12 20:20:18,438] INFO Successfully elected the new master: {"host":"<MYHOST>","port":8081,"master_eligibility":true,"version":1} (io.confluent.kafka.schemaregistry.zookeeper.ZookeeperMasterElector:80)
[2016-09-12 20:20:18,579] INFO jetty-8.1.16.v20140903 (org.eclipse.jetty.server.Server:272)
Sep 12, 2016 8:20:19 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.6 2014-02-18 21:52:53...
[2016-09-12 20:20:19,794] INFO Started [email protected]:8081 (org.eclipse.jetty.server.AbstractConnector:338)
[2016-09-12 20:20:19,795] INFO Server started, listening for requests... (io.confluent.kafka.schemaregistry.rest.Main:44)
[2016-09-12 20:20:31,520] INFO HV000001: Hibernate Validator 5.0.0.Final (org.hibernate.validator.internal.util.Version:27)
[2016-09-12 20:20:31,875] INFO<MY_CLIENT_IP> - - [12/Sep/2016:20:20:31 +0300] "GET /config HTTP/1.1" 200 33  493 (io.confluent.rest-utils.requests:77)
[2016-09-12 20:20:31,947] INFO<MY_CLIENT_IP> - - [12/Sep/2016:20:20:31 +0300] "GET /subjects/ HTTP/1.1" 200 790  532 (io.confluent.rest-utils.requests:77)
[2016-09-12 20:21:18,936] INFO<MY_CLIENT_IP> - - [12/Sep/2016:20:21:18 +0300] "GET /config HTTP/1.1" 200 33  17 (io.confluent.rest-utils.requests:77)
[2016-09-12 20:21:19,141] INFO<MY_CLIENT_IP> - - [12/Sep/2016:20:21:19 +0300] "GET /subjects/ HTTP/1.1" 200 790  88 (io.confluent.rest-utils.requests:77)

[andmarios]

Your schema registry for some reason does not return CORS headers. If you notice, the CORS configuration values you set does not show in your logs.
Just to make sure, we use confluent-2.0.1-2.11.7 and start schema registry like:

path/to/installation/bin/schema-registry-start path/to/installation/etc/schema-registry/schema-registry.properties

A response with CORS headers would be like:

* TCP_NODELAY set
* Connected to schema-registry-201.demo.landoop.com (X.X.X.X) port 28081 (#0)
> OPTIONS / HTTP/1.1
> Host: schema-registry-201.demo.landoop.com:28081
> User-Agent: curl/7.50.2
> Accept: */*
> Origin: https://landoop.com
> Access-Control-Request-Method: POST
> Access-Control-Request-Headers: X-Requested-With
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Mon, 12 Sep 2016 17:39:05 GMT
Date: Mon, 12 Sep 2016 17:39:05 GMT
< Access-Control-Allow-Origin: https://landoop.com
Access-Control-Allow-Origin: https://landoop.com
< Access-Control-Allow-Credentials: true
Access-Control-Allow-Credentials: true
< Access-Control-Max-Age: 1800
Access-Control-Max-Age: 1800
< Access-Control-Allow-Methods: GET,POST,HEAD
Access-Control-Allow-Methods: GET,POST,HEAD
< Access-Control-Allow-Headers: X-Requested-With,Content-Type,Accept,Origin
Access-Control-Allow-Headers: X-Requested-With,Content-Type,Accept,Origin
< Content-Type: application/vnd.sun.wadl+xml
Content-Type: application/vnd.sun.wadl+xml
< Allow: HEAD,POST,GET,OPTIONS
Allow: HEAD,POST,GET,OPTIONS
< Last-Modified: Mon, 12 Sep 2016 19:39:05 CEST
Last-Modified: Mon, 12 Sep 2016 19:39:05 CEST
< Content-Length: 1134
Content-Length: 1134
< Server: Jetty(9.2.12.v20150709)
Server: Jetty(9.2.12.v20150709)

<
* Excess found in a non pipelined read: excess = 1134 url = / (zero-length body)
* Curl_http_done: called premature == 0
* Connection #0 to host schema-registry-201.demo.landoop.com left intact

And starting schema registry, should print values for access.control.allow.origin.

Sep 12 14:28:18 systemd[1]: Starting Confluent Schema Registry Service...
Sep 12 14:28:18 schema-registry-start[36441]: SLF4J: Class path contains multiple SLF4J bindings.
Sep 12 14:28:18 schema-registry-start[36441]: SLF4J: Found binding in [jar:file:/opt/confluent-2.0.1/share/java/confluent-common/slf4j-log4j12-1.7.6.jar!/org/slf4j/im
Sep 12 14:28:18 schema-registry-start[36441]: SLF4J: Found binding in [jar:file:/opt/confluent-2.0.1/share/java/schema-registry/slf4j-log4j12-1.7.6.jar!/org/slf4j/imp
Sep 12 14:28:18 schema-registry-start[36441]: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Sep 12 14:28:18 schema-registry-start[36441]: SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Sep 12 14:28:18 schema-registry-start[36441]: [2016-09-12 14:28:18,933] INFO SchemaRegistryConfig values:
Sep 12 14:28:18 schema-registry-start[36441]: metric.reporters = []
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.connection.url = localhost:22181/confluent
Sep 12 14:28:18 schema-registry-start[36441]: avro.compatibility.level = backward
Sep 12 14:28:18 schema-registry-start[36441]: debug = false
Sep 12 14:28:18 schema-registry-start[36441]: shutdown.graceful.ms = 1000
Sep 12 14:28:18 schema-registry-start[36441]: response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.commit.interval.ms = -1
Sep 12 14:28:18 schema-registry-start[36441]: response.mediatype.default = application/vnd.schemaregistry.v1+json
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.topic = _schemas
Sep 12 14:28:18 schema-registry-start[36441]: metrics.jmx.prefix = kafka.schema.registry
Sep 12 14:28:18 schema-registry-start[36441]: access.control.allow.origin = *
Sep 12 14:28:18 schema-registry-start[36441]: port = 28081
Sep 12 14:28:18 schema-registry-start[36441]: request.logger.name = io.confluent.rest-utils.requests
Sep 12 14:28:18 schema-registry-start[36441]: metrics.sample.window.ms = 30000
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.zk.session.timeout.ms = 30000
Sep 12 14:28:18 schema-registry-start[36441]: master.eligibility = true
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.topic.replication.factor = 3
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.timeout.ms = 500
Sep 12 14:28:18 schema-registry-start[36441]: host.name = schema-registry-201.demo.landoop.com
Sep 12 14:28:18 schema-registry-start[36441]: schema.registry.zk.namespace = schema_registry
Sep 12 14:28:18 schema-registry-start[36441]: kafkastore.init.timeout.ms = 60000
Sep 12 14:28:18 schema-registry-start[36441]: metrics.num.samples = 2
Sep 12 14:28:18 schema-registry-start[36441]: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:135)
Sep 12 14:28:19 schema-registry-start[36441]: [2016-09-12 14:28:19,367] INFO Initialized the consumer offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaSto
Sep 12 14:28:19 schema-registry-start[36441]: [2016-09-12 14:28:19,679] WARN Creating the schema topic _schemas using a replication factor of 1, which is less than th
Sep 12 14:28:19 schema-registry-start[36441]: [2016-09-12 14:28:19,765] INFO [kafka-store-reader-thread-_schemas], Starting  (io.confluent.kafka.schemaregistry.storag
Sep 12 14:28:19 schema-registry-start[36441]: [2016-09-12 14:28:19,910] INFO Wait to catch up until the offset of the last message at 0 (io.confluent.kafka.schemaregi
Sep 12 14:28:20 schema-registry-start[36441]: [2016-09-12 14:28:20,043] INFO Created schema registry namespace localhost:22181/schema_registry (io.confluent.kafka.sch
Sep 12 14:28:20 schema-registry-start[36441]: [2016-09-12 14:28:20,058] INFO Successfully elected the new master: {"host":"schema-registry-201.demo.landoop.com","port":28081,"maste

[jramapuram]

Gah, looks like I'm still on 1.0.1-2!
Weird that it says confluent-platform-2.11.5 though:

> dpkg -l confluent-platform*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                          Version                             Architecture                        Description
+++-=============================================================-===================================-===================================-================================================================================================================================
ii  confluent-platform-2.11.5                                     1.0.1-2                             all                                 Confluent Platform

[andmarios]

2.11.5 refers to the Scala version used to compile the release. Though we don't support Confluent 1.x, we will soon make a release of the docker image where you'll have the choice to proxy the schema-registry through the container's webserver. This could solve your CORS issue if schema registry 1.x's api is compatible with the newer versions. I'll give you a ping here once we push it, maybe later today or tomorrow.

As a side note, If you can upgrade your installation I'd advise you to do it. Confluent has a fast pace adding features to its platform and unless you are on a production system with strict deployment and downtime requirements, you will be better off choosing the latest version.

[andmarios]

Thanks for reporting back. Glad it worked!