Add Confluent Schema Registry compatible API to enable migration about registry

[michaelandrepearce]

VersionedStorable adding a globally unique id to this in the tables/storage would then be able from a single unique id, retrieve a vesionedstorable , which then using this retrieved object can use the root entity to get back the schemametadata also

[michaelandrepearce]

On checking it seems there already is a unique id named "id" for SchemaVersionStorable :), could just use this.

[michaelandrepearce]

id = globally unique id for a schema aka this could be SchemaVersionStorable id.

methods/apis needed to implement that Confluent Avro Serializer/Deserilizer invokes.

Serializer:
int register(subject, schema)

POST /subjects/{subject}/versions
body:
{"schema": "{\"type\": \"string\"}"}

Deserializer:
Schema getBySubjectAndID(subject, id) actually delegates for remote call ->SchemaString getByID(id)

GET /schemas/ids/{id}

int getVersion(subject, schema) actually delegates for remote call -> Schema lookUpSubjectVersion(subject, schema)

POST /subjects/{subject}
body:
{"schema": "{\"type\": \"string\"}"}

[michaelandrepearce]

@satishd We have this now implemented with tests locally ready to share/raise PR for, it builds on some of the other PR's we have open though, to avoid nasty merge conflict issues, any possibility having merged our current PR's

and then ill sort the merge conflicts on this one:

#125

[satishd]

@michaelandrepearce I am fine with having confluent schema registry compatible API but I would like to know your plans on migration path.
We would expect users would need to eventually change ser/des on their existing applications. New application can start this registry ser/des and respective APIs.

[michaelandrepearce]

@satishd with having this will mean we can use confluent serdes, so on day one its just a service switch out, as in we can re-route via networking or quick runtime property change, no need for code deploys, which in a large environment would be almost impossible. the selling point is to have a global schema registry (one source of truth)

We will have for a long time the need to run with confluent protocol on some flows, (0x0 + int id + serialised data), thus my suggestion (issue raised) we adopt this also at client level for produce/consume, so we can migrate clients serdes easier as it also would be compatible, reason why is there is a large eco-system already supporting this. If not any such any flow still not able to migrate will have to remain on the confluent serdes until support is added. Which would hinder adoption.

Areas that would cause this for us as current are:

• Language Support
  ** C++ clients
  ** Python clients

• ThirdParty Tools/Systems e.g.
  ** Vertica
  ** StreamSets

[satishd]

@michaelandrepearce That makes sense. It would be great if you can write a doc or wiki about the migration path that you are following, it will be useful to other users.

[michaelandrepearce]

@satishd will do, and happy to contribute this, could you possibly suggest a sensible location for us to make this wiki page.

[satishd]

@michaelandrepearce You can add as a new wiki page here for now.

[harshach]

@michaelandrepearce @satishd it would be good to have this in docs here https://github.com/hortonworks/registry/tree/master/docs and they will be published once we merge here http://registry-project.readthedocs.io
But whichever makes it easy to get this in, if wiki is easier lets do that. This is been most asked question so far.

[michaelandrepearce]

Thanks @harshach @satishd , i have raised a PR #164 for initial review, its not 100% so not quite ready for merge (e.g. need to adjust commit messages etc), but if you could read over, cheers.

[satishd]

@michaelandrepearce ConfluentSchemaRegistryCompatibleResource and confluent protocol ser/des addresses compatible required for migration from confluent SR. Do we have anything remaining for migration w.r.t compatible API? If not, we can close this issue.

[michaelandrepearce]

@satishd i think so, anything else can just be a new issue. Ill close it now.