stargate / stargate

An open source data gateway

Stargate

An open source data API gateway.

Stargate is a data gateway deployed between client applications and a Cassandra database. For developers, it makes it easy to use Cassandra for any application workload by adding plugin support for new APIs, data types, and access methods - even secondary database models. It's built with extensibility as a first-class citizen to enable rapid innovation. For operators, Stargate introduces microservice architecture, allowing independent deployment and scale of storage nodes, API Service nodes, and coordinator nodes in Cassandra clusters.

• For quick instructions on how to bring up Stargate on your desktop using Docker, check out the Docker compose instructions.
• If you wish to deploy in Kubernetes, there is also a Helm chart you can use to install Stargate alongside an existing Cassandra cluster, or visit the K8ssandra project for a distribution that packages Cassandra, Stargate, and additional tooling.
• For more information about how to deploy and use Stargate, visit stargate.io
• To learn how to participate in our community, visit our community page
• To set up and use a Stargate development environment, see the instructions for the coordinator and APIs.

Contents

• Introduction
• Repositories
• Issue Management

Introduction

We created Stargate because we got tired of using different databases and different APIs depending on the work that we were trying to get done. With "read the manual" fatigue and lengthy selection processes wearing on us every time we started a new project, we thought - why not create a framework that can serve many APIs for a range of workloads?

This project enables customization of all aspects of data access and has modules for authentication, APIs, request handling / routing, and persistence backends.
The current form is specific to Apache Cassandra (C*) compatible backends.

As shown in the figure below, Stargate is often deployed behind a load balancer or proxy and exposes multiple endpoints to client applications, including HTTP APIs, gRPC, and the Cassandra Query Language (CQL). Stargate sits in front of a Cassandra cluster which is used as the storage backend.

Stargate consists of the following components, which we introduce briefly here with links to the corresponding modules in this monorepo.

API Services

These are independently scalable microservices which various APIs, typically HTTP based. These modules can be found under the apis directory:

• sgv2-restapi: API implementation for exposing Cassandra data over REST
• sgv2-graphqlapi: API implementation for exposing Cassandra data over GraphQL
• sgv2-docsapi: API implementation for exposing Cassandra data over a Document API

Each API Service contains its own integration test suite that tests it against the coordinator node and supported Cassandra backends. There is also a sgv2-quarkus-common module containing utilities that may be used by all Java/Quarkus based API services.

Coordinator Node

Coordinator nodes participate as non-data storing nodes in the backing Cassandra cluster, which enables them to read and write data more efficiently. Stargate Coordinator nodes can also be scaled independently. Coordinator nodes expose gRPC and CQL interfaces for fast access by client applications. The following are the key modules comprising the coordinator and its exposed interfaces:

• core: Common classes used throughout the other coordinator modules
• cql: API implementation for the Cassandra Query Language
• grpc: fast CQL over gRPC implementation (HTTP-based interface equivalent to CQL performance)
• bridge: gRPC-based interface used by API services
• health-checker: HTTP endpoints useful for health checking coordinator nodes
• metrics-jersey: metrics collection for the coordinator node and its exposed interfaces
• stargate-starter: the main Java application used to start the coordinator via the starctl script

Persistence Services

Stargate coordinator nodes support a pluggable approach for implementing the coordination layer to execute requests passed by API services and other interfaces to underlying data storage instances. Persistence service implementations are responsible handling and converting requests to database queries, dispatching to a specific version of Cassandra, and returning and serving responses.

• persistence-api: Interface for working with persistence services
• persistence-common: Utilities shared by the persistence services
• persistence-cassandra-3.11: Joins C* 3.11 cluster as coordinator-only node (does not store data) mocks C* system tables for native driver integration, executes requests with C* storage nodes using C* QueryHandler/QueryProcessor, converts internal C* objects and ResultSets to Stargate Datastore objects.
• persistence-cassandra-4.0: (same as above but for Cassandra 4.0)
• persistence-dse-6.8: (same as above but for DataStax Enterprise 6.8)

Authentication and Authorization Services

Stargate coordinator nodes also support a pluggable authentication and authorization approach.

• authnz: Interface for working with auth providers
• auth-api: REST service for generating auth tokens
• auth-table-based-service: Service to store tokens in the database
• auth-jtw-service: Service to authenticate using externally generated JSON Web Tokens (JWTs)

Coordinator Node Testing

The following modules provide support for testing:

• testing: Integration test suite for the coordinator node modules
• testing: Testing helpers
• persistence-test: Common utilities for testing persistence services

Instructions for running and extending the test suite can be found in the developer guide.

Repositories

Here is an overview of the key repositories in the Stargate GitHub organization:

• stargate/stargate: This repository is the primary entry point to the project. It is a monorepo containing all of the Stargate modules
• stargate/docs: This repository contains the user docs hosted on stargate.io
• stargate/website: This repository contains the code for the website hosted on stargate.io

The organization also contains several gRPC client libraries for various languages.

Issue Management

You can reference the CONTRIBUTING.md for a full description of how to get involved, but the short of it is below.

• If you've found a bug (use the bug label) or want to request a new feature (use the enhancement label), file a GitHub issue
• If you're not sure about it or want to chat, reach out on our Discord
• If you want to write some user docs 🎉 head over to the stargate/docs repo, Pull Requests accepted!

Supported Versions and Branching Strategy

The Stargate project maintains support for the current major version number and one major version number previous. We mark the previous major version number as deprecated in order to encourage usage of the latest version. We anticipate supporting a major version number release N for one year after the subsequent major version number release (N+1), or until the major version number release (N+2), whichever comes first. For example, v1 will be supported until when Stargate v3 is released, but no later than October 2023 (1 year after release date of Stargate v2). We anticipate producing a new major version number release every 6-12 months.

Supporting a release entails making bug fixes, keeping dependencies up to date, and making sure Docker images are free from vulnerabilities.

The current major version is maintained on the default main branch. The prior major version number is maintained on a version branch, for example, v1.

We make bug fixes in all supported releases. The recommended approach is to commit a fix first to the previous major version branch (if applicable) and merge it forward into the current major version branch.

We use minor version numbers to indicate any changes to the coordinator that would cause compatibility changes for Stargate APIs. For example, a breaking change to the bridge API in v2.0.x would cause creation of a v2.1.0 release, and any API implementations would need to explicitly update in order to be compatible. You can use any version of the coordinator and API that have same the major and minor version number without issues.

We iterate forward rather than producing patch releases. For example, for a vulnerability found in v2.0.3, we'd make any required fixes and dependency updates and release v2.0.4. We maintain a regular release cadence of approximately twice a month but can iterate more quickly as the situation dictates.

Thanks

This project uses tools provided by YourKit, LLC. YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.

stargate's Issues

Rest Api V2: not able to handle integer in post request

curl --request POST \
  --url 'localhost:8082/v2/keyspaces/a_manu/widgets' \
 --header 'x-cassandra-token: ef15d2dc-0741-4d58-a499-3ada37788d51'  --header 'accepts: application/json' \
--header 'content-type: application/json' \
--data '{"hour_created":"2020-01-24T11:00:00.00Z","machine_code":"131", "serial_number":"afadfaf","times_inspected":1}' 
{"description":"Server error: java.lang.Integer cannot be cast to java.lang.String","code":500}%

start of the stack

CREATE TABLE a_manu.widgets (
    hour_created timestamp,
    machine_code text,
    serial_number text,
    created_at timestamp,
    est_unit_cost decimal,
    est_unit_cost_updated timestamp,
    factory_code text,
    inspection_notes text,
    last_inspected_at timestamp,
    part_name text,
    part_number text,
    times_inspected int,
    PRIMARY KEY ((hour_created, machine_code), serial_number)
) WITH CLUSTERING ORDER BY (serial_number ASC)

ERROR [2020-09-14 14:15:39,471] io.stargate.web.resources.RequestHandler: Error when executing request
! java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.String
! at io.stargate.web.resources.Converters.colToValue(Converters.java:249)
! at io.stargate.web.resources.v2.RowsResource.lambda$null$2(RowsResource.java:150)
! at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
! at java.util.Iterator.forEachRemaining(Iterator.java:116)
! at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
! at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
! at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
! at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
! at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
! at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
! at io.stargate.web.resources.v2.RowsResource.lambda$add$3(RowsResource.java:151)
! at io.stargate.web.resources.RequestHandler.handle(RequestHandler.java:24)
! at io.stargate.web.resources.v2.RowsResource.add(RowsResource.java:141)
! at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
! at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
! at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
! at java.lang.reflect.Method.invoke(Method.java:498)

There is a hack workaround for this, you can just send the integer as a string and it will work, but this will trip a lot of folks up.

Stargate nodes have inconsistent view of their peers #github

Stargate nodes that are using another stargate node as cluster seed don't have all the topology information from other Stargate nodes, which limits the number of nodes that will be used as coordinator, unless the right stargate node is used as contact point:
[file id="5e6a0524-2395-4315-b07e-1429f520af1d"]

Don't ignore clustering key on get/delete row

Check that all partition keys are specified and then clustering keys are optional. It is possible to have some clustering keys and not all, but in the order they are specified.

So, if the primary key is PRIMARY KEY ((pkey1, pkey2), ccol1, ccol2, ccol3) you have to specify pkey1 and pkey2, but you don't have to specify any clustering columns. If you specify ccol2 you need to have also specified ccol1 and if you specified ccol3 you need to have specified ccol1 and ccol2.

Support running integration tests on macOS

We should support running integration tests on macOS w/ or w/o test containers (i.e. use ccm directly for the backend cluster).

REST v1 /query errors 500 when unexpected JSON data key is used

http POST http://$STARGATE:8082/v1/keyspaces/ks1/tables/table8/rows/query X-Cassandra-Token:$TOKEN unknownparam=""
HTTP/1.1 500 Internal Server Error
Content-Length: 47
Content-Type: application/json
Date: Mon, 14 Sep 2020 12:25:55 GMT

{
    "code": 500, 
    "description": "Server error: null"
}

I think it should either be ignored or return some 4xx.

Performance test CQL

Some interesting ideas what to expect / measure and when we should see benefits:https://www.youtube.com/watch?v=K0sQvaxiDH0

Lacking Acceptance Criteria.Probably to be decomposed into multiple tickets.

Add integration tests for auth-api

Currently the only testing of the auth-api is through REST and Graphql so we need dedicated tests for this module. Please create tests in a similar manner to what currently exists for REST in the testing module that exercises functionality of all endpoints.

Create low-level interface over the top of C*/DSE

Currently, we're using DataStore as our interface over the top of C*/DSE. This is a fairly high-level interface that requires somewhat non-trivial implementations. If we intend to support multiple versions of C* and DSE it might be better support a lower-level interface that has a much lower code footprint per version supported. With that lower-level interface in place it may be possible to have a single implementation of DataStore.

CI: Create tests using multiple stargate instances

Create tests using multiple stargate instances running locally.

That way we can test peers and schema metadata among other things.

Add core API for password authentication

BothcqlshandJDBC(SQL) clients ask for username/password on the client side, but there's no authentication API defined in Stargate core that allows validating passwords (without creating tokens).

It would be nice to add an API specifically for validating username/password combinations.

Support schema changes for the graphql endpoint

Preface: In the legacy Go implementation, all schema changes occur on its own special endpoint called graphql-schema and only allows for keyspaces and tables creation. This ticket is to address schema changes.

The resolution should have the follow attributes:

Follow best practices in database schema changes

In a multi team environment where each team is given their own keyspace, users should be able to manage their own tables without conflict.

A user should be able to apply their development changes to a production instance

Be able to Add, remove, and rename table fields

Be able to add fields to a user defined types

Be able to remove user defined types

Schemas and changes should be able to be tracked in git

Some notes:

Users may feel uncomfortable having a schema alter endpoint live next to their mission critical apis

Many developers will want the option to rollback a change if the application encounters an issue

Proposed User Journey:

1. Download your current schema
2. Make changes
3. Validate the schema changes with the endpoint, a list of schema change operations is returned
4. Migrate the schema by posting the list of schema changes

Key features:

Schema changes are done by a diff

Renaming a column can be done through a @rename(field: "name") attribute.

The list of schema change operations can be committed, along with the schema

Notes:

This process does not account for rollback or staging of changes due to lack of statefulness of the current graphql architecture.

Use 3-node cluster as test backend

We need add integration tests against multiple C* and DSE nodes.

There are issues that are only likely to appear in a multi-node cluster: consistency; schema mismatches; ...

┆Issue is synchronized with this Asana task by Unito

Accepting CQL connection prematurely

In multiple failing dtests, the test restarts at least 1 C* node, then immediately performs query that fails with "Cannot achieve consistency level".

After bouncing a C* node, the test code tells the driverwait_other_notice=True, wait_for_binary_proto=True- thiswouldensure that the bounced node is available before proceeding,if it were connecting directly to C*. But with the driver connecting to Stargate, it does not wait long enough before responding to the connection request. I can see in the Stargate log that it processes the failed query immediately after the handshake with the bounced node butbeforelogging that the bounced node is "UP". So, the query fails with insufficient live replicas.

I verified that this is a timing issue by adding a sleep in test code. I can get failing tests to consistently pass by waiting between driver connection and first query using that connection.

While this could be resolved in tests by adding retry loops or sleeps, it seems like it will be a user experience issue also.There are 2 scenarios here:

1-node C* cluster with 1 SG node - this seems straightforward ... SG shouldn't finish a new connection request with{{wait_other_notice=True, wait_for_binary_proto=True}} until the C* node is fully UP.

multi-node C* cluster with 1 SG node - seems harder since the client no longer connects directly to the node being bounced ... how can the client know when that node is fully UP?

Rest API: limit request and parameter sizes

Should be:

Configurable with reasonable defaults (ie not unlimited)

cover total request size as well as per parameter limits

Throw a nice error message that is easy for the user to understand what is going on without bleeding too much information.

Log an error in the server logs with some form of no spamlogger to keep from inundating the logs with bad requests.

Would like:

adjustable per parameter

lower defaults for small thing that are always small and high defaults for things that are often big (where clauses come to mind)

StarterTest - SetStargatePropertiesWithBadHostSeedNode

Seeing this issue on Windows running Java 14. This test fails on an expected exception for the incorrect seed list entry of "datasta.cmo". All other tests are running successful for the StarterTest.

org.opentest4j.AssertionFailedError: Expected setStargateProperties() to throw RuntimeException ==> Expected java.lang.RuntimeException to be thrown, but nothing was thrown.

at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:71)
at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:41)
at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:2968)
at io.stargate.starter.StarterTest.testSetStargatePropertiesWithBadHostSeedNode(StarterTest.java:52)

Rest Api: Better Primary Key Validation

Should:

Explain to the user what key is missing or what operation is not allowed

Validate all of the partition key is met and not using gt, lt or anything that wouldn't be a valid CQL query anyway.

Would like:

it to be thrown before processing any other parts of the request.

Add integration tests for CQL

Load selected settings from a cassandra.yaml

There are a number of C* features that do not work due to a mismatch in settings between Stargate and the backend. Several of these have been encountered when executing the cassandra-dtest suite via Stargate:

authentication (and an error should be thrown if C2 is started with--enable-authbut cassandra.yaml doesn't agree)

internode SSL

user-defined functions

partitioner

unlogged_batch_across_partitions_warn_threshold

more?

Not only will this config loading allow us to run more existing C* tests, it will also unblock OSS users who want to use Stargate with clusters that have some non-default settings.

┆Issue is synchronized with this Asana task by Unito

Stargate metrics

Initial set of questions to be answered by metrics:

are stargate nodes uniformly loaded with requests

is there any "slow" node in cluster, what are per node latencies ?

Seems like first thing would be requests handling histograms.For all APIs.

Consider defaulting to ScyllaDB over Cassandara

Scylla evidently offers significantly better performance, while also being a wide store. Please consider this independent benchmark as of 2020:

https://www.datasciencecentral.com/profiles/blogs/scylla-vs-cassandra-performance-comparison

Or is latency not important to this project? Why should users have to pay more to get the same performance?

Graphql Enhancements

Persisted Queries for graphql

Exposing a fully flexible query engine to the general internet requires considerable thought and planning than exposing purpose-built apis. Persisted queries allow specific queries and mutations to be defined in graphql. It also improves performance considerably and helps facilitate backwards-compatible schema changes.

Additionally, persisted queries can be an entry point to support c* atomicity since it can perform multiple graphql requests in a single call.

We should also allow users the ability to sculpt their api definition so they can handle schema migration situations themselves.

https://www.apollographql.com/blog/improve-graphql-performance-with-automatic-persisted-queries-c31d27b8e6ea

@api(path: "v1/cartByUser")
query userCartEntries($userId: Uuid!, $pageState: String) {
  cartByUser(value: {userId: $userId}, pageState: $pageState) {
    values {
      cartId
      productName
      quantity
    }
    pageState
  }
}

$ curl -X GET http://astra/graphql/cart/v1/cartByUser?userId=0000-0000-0000-0000

Stargate runs out of memory during Java driver duration test

I tried running the Java driver duration tests against a Stargate cluster. After approximately 10 minutes and a few hundred thousand requests, Stargate drops all native connections, stops accepting new ones, and logs the following:

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007efbc7280000, 207618048, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 207618048 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/automaton/hs_err_pid9232.log

Those duration tests run a simple scenario (insert a row in a table, read it back, delete it) in a loop, from many different threads. The sources arehere. Here is aFallout scriptto set up an environment and launch them.

I wasn't able to reproduce the issue locally.

┆Issue is synchronized with this Asana task by Unito

Create common backing services for REST v1 and v2

Much of the logic between v1 and v2 of the REST API is shared so rather than duplicating in each resource we should instead create common objects for them to interact with. For example, rather than duplicating the same code in io.stargate.web.resources.v2.schemas.ColumnsResource and io.stargate.web.resources.ColumnResource instead create a SchemaService (or something to that effect) that will allow for interacting with persistence and returning the relevant values

testing integrations

Hot deploy should work for bundles

Tried to hot deploy the restapi and it ended up bringing everything else down.

Remove java-driver dependency from `DataStore` interface

Improve developer experience

Improve docs around building

Speed up maven build

Debugging

Running cassandra

Iterating on modules without restarting persistence

CQL: Page state is not set in response

The page state is not included in the response when there are further pages.

add http/2 support to the rest API

According to the docs this looks supported out of the box in dropwizard and in the underlying jetty one can configure both http 1.1 and http/2 at the same time, so I'm assuming it's easy enough to do with drop wizard.

Look athttps://www.dropwizard.io/en/latest/manual/configuration.htmlunder HTTP/2 over TLSand HTTP/2 Plain Text.

Ideally we'd just turn on the correct mode depending on if SSL is enabled or not and always turn on HTTP/2 and HTTP 1.1 to accommodate the widest variety of clients.

Add tests for auth-api

Support tuples and UDTs in RestAPI

Stargate should not waste memory on ChunkCache (or other default storage related buffers)

Present in the logs:

16:09:21.546 [MemtableFlushWriter:3] INFO org.apache.cassandra.cache.ChunkCacheImpl - Created chunk cache of size 27.022GiB, allocating page-aligned buffers from cached reads pool except for sizes >= 256.000KiB

Now the tricky part is that if we create too small ChunkCache then AIO will get disabled by default?

So we should find a away to:

not waste memory on ChunkCache

but also do not disable AIO

Rest Api: Find a way to shrink paging state

In the Rest Api paging state is supported and passed base64 encoded. This brings up some concerns about top end parameter size depending on the environment Stargate is deployed in (this varies depending on configuration), another consideration is the protocol version changes the format. Possible fixes for this are storing paging state and reference by id, compressing the paging state first to help minimise running into limits . We should address this or have a plan for this before getting too many users. There are several data models (the typical monitoring service pattern) that result in very large partition keys and since that's part of the state stored in paging state this will cause us to blow up several http server limits. Since this will affect API design and implementation I suggest we at least start thinking about this now.

http POST http://$STARGATE:8081/v1/auth without body results in 500 and NPE

HTTP/1.1 500 Server Error
Cache-Control: must-revalidate,no-cache,no-store
Connection: close
Content-Length: 41
Content-Type: application/json;charset=utf-8
Server: Jetty(9.4.24.v20191120)

{
    "description": "Unexpected error occurs"
}

In the stargate logs:

09:23:35.797 [qtp231500219-152] WARN org.eclipse.jetty.server.HttpChannel - /v1/auth
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: java.lang.NullPointerException
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:310)
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:264)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 16 common frames omitted
Caused by: java.lang.NullPointerException: null
	at io.stargate.auth.api.AuthResource.createToken(AuthResource.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 37 common frames omitted
09:23:35.798 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - thrownException s=HANDLING rs=BLOCKING os=OPEN is=READY awp=false se=true i=true al=0
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: java.lang.NullPointerException
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:310)
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:264)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 16 common frames omitted
Caused by: java.lang.NullPointerException: null
	at io.stargate.auth.api.AuthResource.createToken(AuthResource.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 37 common frames omitted
09:23:35.798 [qtp231500219-152] WARN org.eclipse.jetty.server.HttpChannelState - unhandled due to prior sendError
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:500)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:135)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: java.lang.NullPointerException
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
	at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:310)
	at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:264)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 16 common frames omitted
Caused by: java.lang.NullPointerException: null
	at io.stargate.auth.api.AuthResource.createToken(AuthResource.java:70)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 37 common frames omitted
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@4b8c17a2{s=HANDLING rs=BLOCKING os=OPEN is=READY awp=false se=true i=true al=0}
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - nextAction(false) SEND_ERROR HttpChannelState@4b8c17a2{s=HANDLING rs=BLOCKING os=OPEN is=READY awp=false se=false i=false al=0}
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - action SEND_ERROR HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=HANDLING rs=BLOCKING os=OPEN is=READY awp=false se=false i=false al=0},r=1,c=false/false,a=HANDLING,uri=//10.101.37.135:8081/v1/auth,age=8}
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.servlet.ErrorPageErrorHandler - getErrorPage(POST /v1/auth) => error_page=null (from global default)
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@4b8c17a2{s=HANDLING rs=BLOCKING os=OPEN is=READY awp=false se=false i=false al=0}
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - nextAction(false) COMPLETE HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETING os=OPEN is=READY awp=false se=false i=false al=0}
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - action COMPLETE HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETING os=OPEN is=READY awp=false se=false i=false al=0},r=1,c=false/false,a=HANDLING,uri=//10.101.37.135:8081/v1/auth,age=8}
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - sendResponse info=null content=DirectByteBuffer@6823b0cd[p=0,l=41,c=32768,r=41]={<<<{"description":"Unexpected error occurs"}>>>"}\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00} complete=true committing=true callback=org.eclipse.jetty.util.Callback$4@5c2e2020
09:23:35.799 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - COMMIT for /v1/auth on HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETING os=COMMITTED is=READY awp=false se=false i=false al=0},r=1,c=false/false,a=HANDLING,uri=//10.101.37.135:8081/v1/auth,age=8}
500 null HTTP/1.1
Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json;charset=utf-8


09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - generate: NEED_HEADER for org.eclipse.jetty.server.HttpConnection$SendCallback@69ff81fa[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@6d383d2c] (null,[p=0,l=41,c=32768,r=41],true)@START
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.http.HttpGenerator - generateHeaders HTTP/1.1{s=500,h=2,cl=-1} last=true content=DirectByteBuffer@6823b0cd[p=0,l=41,c=32768,r=41]={<<<{"description":"Unexpected error occurs"}>>>"}\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.http.HttpGenerator - Cache-Control: must-revalidate,no-cache,no-store
Content-Type: application/json;charset=utf-8


09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.http.HttpGenerator - EOF_CONTENT
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - generate: FLUSH for org.eclipse.jetty.server.HttpConnection$SendCallback@69ff81fa[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@6d383d2c] ([p=0,l=197,c=8192,r=197],[p=0,l=41,c=32768,r=41],true)@COMPLETING
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.io.WriteFlusher - write: WriteFlusher@1103c24f{IDLE}->null [HeapByteBuffer@4b4400dd[p=0,l=197,c=8192,r=197]={<<<HTTP/1.1 500 Server Error...y(9.4.24.v20191120)\r\n\r\n>>>-Length: ...\x00\x00\x00\x00\x00\x00\x00},DirectByteBuffer@6823b0cd[p=0,l=41,c=32768,r=41]={<<<{"description":"Unexpected error occurs"}>>>"}\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00}]
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.io.WriteFlusher - update WriteFlusher@1103c24f{WRITING}->null:IDLE-->WRITING
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.io.ChannelEndPoint - flushed 238 SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OPEN,fill=-,flush=W,to=11/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4dc61483{s=COMPLETING}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETING os=COMMITTED is=READY awp=false se=false i=false al=0},r=1,c=false/false,a=HANDLING,uri=//10.101.37.135:8081/v1/auth,age=9}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.io.WriteFlusher - Flushed=true written=238 remaining=0 WriteFlusher@1103c24f{WRITING}->null
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.io.WriteFlusher - update WriteFlusher@1103c24f{IDLE}->null:WRITING-->IDLE
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - generate: SHUTDOWN_OUT for org.eclipse.jetty.server.HttpConnection$SendCallback@69ff81fa[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@6d383d2c] ([p=197,l=197,c=8192,r=0],[p=41,l=41,c=32768,r=0],true)@END
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - generate: DONE for org.eclipse.jetty.server.HttpConnection$SendCallback@69ff81fa[PROCESSING][i=HTTP/1.1{s=500,h=2,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$SendCallback@6d383d2c] ([p=197,l=197,c=8192,r=0],[p=41,l=41,c=32768,r=0],true)@END
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - completed HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETING os=COMPLETED is=READY awp=false se=false i=false al=0}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.io.AbstractEndPoint - shutdownOutput SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=END,0 of 0},g=HttpGenerator@4dc61483{s=END}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETED os=COMPLETED is=READY awp=false se=false i=false al=0},r=1,c=true/true,a=HANDLING,uri=//10.101.37.135:8081/v1/auth,age=9}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - unhandle HttpChannelState@4b8c17a2{s=HANDLING rs=COMPLETED os=COMPLETED is=READY awp=false se=false i=false al=0}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - nextAction(false) TERMINATED HttpChannelState@4b8c17a2{s=IDLE rs=COMPLETED os=COMPLETED is=READY awp=false se=false i=false al=0}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - action TERMINATED HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=COMPLETED os=COMPLETED is=READY awp=false se=false i=false al=0},r=1,c=true/true,a=IDLE,uri=//10.101.37.135:8081/v1/auth,age=9}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - onCompleted for /v1/auth written=41
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannelState - recycle HttpChannelState@4b8c17a2{s=IDLE rs=COMPLETED os=COMPLETED is=READY awp=false se=false i=false al=0}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.http.HttpParser - close HttpParser{s=END,0 of 0}
09:23:35.800 [qtp231500219-152] DEBUG org.eclipse.jetty.http.HttpParser - END --> CLOSE
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpChannel - !handle TERMINATED HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.ChannelEndPoint - filled 0 HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.ChannelEndPoint - filled 0 HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} filled 0 HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} parse HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00} {}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.http.HttpParser - parseNext s=CLOSE HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} parsed false HttpParser{s=CLOSE,0 of 0}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.AbstractConnection - fillInterested HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=1/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.FillInterest - interested FillInterest@6488611d{AC.ReadCB@d95fd58{HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}}}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.ChannelEndPoint - changeInterests p=false 0->1 for SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=0/30000}{io=0/1,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.ManagedSelector - Queued change org.eclipse.jetty.io.ChannelEndPoint$1@4f924b29 on ManagedSelector@6b6afca9{STARTED} id=7 keys=1 selected=0 updates=0
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.io.ManagedSelector - Wakeup on submit ManagedSelector@6b6afca9{STARTED} id=7 keys=1 selected=0 updates=1
09:23:35.801 [qtp231500219-152] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=0/30000}{io=0/1,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} onFillable exit HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
09:23:35.801 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 woken with none selected
09:23:35.802 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - ran org.eclipse.jetty.io.ManagedSelector$$Lambda$518/1167858937@385e7d67 in QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 woken up from select, 0/0/1 selected
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 processing 0 keys, 1 updates
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - updateable 1
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - update org.eclipse.jetty.io.ChannelEndPoint$1@4f924b29
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 0 -> 1 on SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=0/30000}{io=1/1,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - updates 0
09:23:35.802 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 waiting with 1 keys
09:23:36.113 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 woken up from select, 1/1/1 selected
09:23:36.113 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 processing 1 keys, 0 updates
09:23:36.113 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - selected 1 sun.nio.ch.SelectionKeyImpl@1d132a1b SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=311/30000}{io=1/1,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} 
09:23:36.113 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ChannelEndPoint - onSelected 1->0 r=true w=false for SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=312/30000}{io=1/0,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.114 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ChannelEndPoint - task CEP:SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=312/30000}{io=1/0,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING
09:23:36.114 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=1/16,p=0} tryExecute EatWhatYouKill@7edb2d40/SelectorProducer@464ab738/PRODUCING/p=false/QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=1,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}][pc=0,pic=0,pec=0,epc=1]@2020-09-14T09:23:36.114Z
09:23:36.114 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=0/16,p=0}@3605953b offer EatWhatYouKill@7edb2d40/SelectorProducer@464ab738/PRODUCING/p=false/QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=1,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=0/16,p=0}][pc=0,pic=0,pec=0,epc=1]@2020-09-14T09:23:36.114Z
09:23:36.114 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=0/16,p=1} startReservedThread p=1
09:23:36.115 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - queue ReservedThreadExecutor@76b436fa{s=0/16,p=1}@e005e9 startThread=0
09:23:36.115 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - run ReservedThreadExecutor@76b436fa{s=0/16,p=1}@e005e9 in QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=0/16,p=1}]
09:23:36.115 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=1/16,p=1}@e005e9 started
09:23:36.115 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=1/16,p=0}@e005e9 waiting
09:23:36.115 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.strategy.EatWhatYouKill - EatWhatYouKill@7edb2d40/SelectorProducer@464ab738/IDLE/p=true/QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=0/16,p=1}][pc=0,pic=0,pec=0,epc=1]@2020-09-14T09:23:36.115Z m=EXECUTE_PRODUCE_CONSUME t=CEP:SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=313/30000}{io=1/0,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}:runFillable:BLOCKING/BLOCKING
09:23:36.115 [qtp231500219-216] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=0/16,p=1}@3605953b task=EatWhatYouKill@7edb2d40/SelectorProducer@464ab738/IDLE/p=true/QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}][pc=0,pic=0,pec=0,epc=1]@2020-09-14T09:23:36.115Z
09:23:36.115 [qtp231500219-215] DEBUG org.eclipse.jetty.io.FillInterest - fillable FillInterest@6488611d{AC.ReadCB@d95fd58{HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=FI,flush=-,to=314/30000}{io=1/0,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}}}
09:23:36.116 [qtp231500219-216] DEBUG org.eclipse.jetty.util.thread.strategy.EatWhatYouKill - EatWhatYouKill@7edb2d40/SelectorProducer@464ab738/IDLE/p=true/QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}][pc=0,pic=0,pec=0,epc=2]@2020-09-14T09:23:36.115Z tryProduce true
09:23:36.116 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - updateable 0
09:23:36.116 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - updates 0
09:23:36.116 [qtp231500219-215] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=314/30000}{io=1/0,kio=1,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} onFillable enter HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
09:23:36.116 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ChannelEndPoint - Key interests updated 1 -> 0 on SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=314/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.116 [qtp231500219-215] DEBUG org.eclipse.jetty.io.AbstractEndPoint - shutdownInput SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,OSHUT,fill=-,flush=-,to=314/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.116 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 waiting with 1 keys
09:23:36.116 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ChannelEndPoint - doClose SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=314/30000}{io=0/0,kio=0,kro=1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.117 [qtp231500219-215] DEBUG org.eclipse.jetty.io.WriteFlusher - ignored: WriteFlusher@1103c24f{IDLE}->null
java.nio.channels.ClosedChannelException: null
	at org.eclipse.jetty.io.WriteFlusher.onClose(WriteFlusher.java:521)
	at org.eclipse.jetty.io.AbstractEndPoint.onClose(AbstractEndPoint.java:354)
	at org.eclipse.jetty.io.ChannelEndPoint.onClose(ChannelEndPoint.java:214)
	at org.eclipse.jetty.io.AbstractEndPoint.doOnClose(AbstractEndPoint.java:225)
	at org.eclipse.jetty.io.AbstractEndPoint.shutdownInput(AbstractEndPoint.java:107)
	at org.eclipse.jetty.io.ChannelEndPoint.fill(ChannelEndPoint.java:237)
	at org.eclipse.jetty.server.HttpConnection.fillRequestBuffer(HttpConnection.java:333)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
	at java.lang.Thread.run(Thread.java:748)
09:23:36.117 [qtp231500219-215] DEBUG org.eclipse.jetty.io.FillInterest - onClose FillInterest@6488611d{null}
09:23:36.117 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ManagedSelector - Wakeup ManagedSelector@6b6afca9{STARTED} id=7 keys=1 selected=0 updates=0
09:23:36.117 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - queue org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@2346b0b6 startThread=1
09:23:36.117 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 woken with none selected
09:23:36.117 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 woken up from select, 0/0/0 selected
09:23:36.117 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - Starting Thread[qtp231500219-217,5,main]
09:23:36.117 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 processing 0 keys, 0 updates
09:23:36.117 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - updateable 0
09:23:36.117 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - updates 0
09:23:36.117 [qtp231500219-216] DEBUG org.eclipse.jetty.io.ManagedSelector - Selector sun.nio.ch.EPollSelectorImpl@13d4c611 waiting with 0 keys
09:23:36.118 [qtp231500219-215] DEBUG org.eclipse.jetty.io.ChannelEndPoint - filled -1 HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:36.118 [qtp231500219-215] DEBUG org.eclipse.jetty.http.HttpParser - atEOF HttpParser{s=CLOSE,0 of 0}
09:23:36.118 [qtp231500219-215] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=316/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} filled -1 HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:36.118 [qtp231500219-217] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - Runner started for QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=13<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]
09:23:36.118 [qtp231500219-217] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - run org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@2346b0b6 in QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=13<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]
09:23:36.119 [qtp231500219-215] DEBUG org.eclipse.jetty.io.AbstractEndPoint - close SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=317/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.119 [qtp231500219-215] DEBUG org.eclipse.jetty.io.AbstractEndPoint - close(null) SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=317/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.119 [qtp231500219-217] DEBUG org.eclipse.jetty.io.ManagedSelector - Destroyed SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=317/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.119 [qtp231500219-215] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=317/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} parse HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00} {}
09:23:36.119 [qtp231500219-215] DEBUG org.eclipse.jetty.http.HttpParser - parseNext s=CLOSE HeapByteBuffer@4b4400dd[p=0,l=0,c=8192,r=0]={<<<>>>HTTP/1.1 ...\x00\x00\x00\x00\x00\x00\x00}
09:23:36.119 [qtp231500219-217] DEBUG org.eclipse.jetty.io.AbstractConnection - onClose HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=317/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSE,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.119 [qtp231500219-215] DEBUG org.eclipse.jetty.http.HttpParser - CLOSE --> CLOSED
09:23:36.119 [qtp231500219-217] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - ran org.eclipse.jetty.io.ManagedSelector$DestroyEndPoint@2346b0b6 in QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=13<=200,i=0,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]
09:23:36.119 [qtp231500219-215] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=318/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSED,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} parsed false HttpParser{s=CLOSED,0 of 0}
09:23:36.120 [qtp231500219-215] DEBUG org.eclipse.jetty.server.HttpConnection - releaseRequestBuffer HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=318/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSED,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.120 [qtp231500219-215] DEBUG org.eclipse.jetty.io.AbstractEndPoint - shutdownOutput SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=318/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSED,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0}
09:23:36.120 [qtp231500219-215] DEBUG org.eclipse.jetty.server.HttpConnection - HttpConnection@d95fd58::SocketChannelEndPoint@1b4b7b02{/10.151.0.238:33432<->/10.101.37.135:8081,CLOSED,fill=-,flush=-,to=319/30000}{io=0/0,kio=-1,kro=-1}->HttpConnection@d95fd58[p=HttpParser{s=CLOSED,0 of 0},g=HttpGenerator@4dc61483{s=START}]=>HttpChannelOverHttp@7a7a73bd{s=HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0},r=1,c=false/false,a=IDLE,uri=null,age=0} onFillable exit HttpChannelState@4b8c17a2{s=IDLE rs=BLOCKING os=OPEN is=IDLE awp=false se=false i=true al=0} null
09:23:36.121 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=2/16,p=0}@16804601 waiting
09:23:47.562 [OptionalTasks:1] DEBUG o.a.c.db.SizeEstimatesRecorder - Node is not part of the ring; not recording size estimates
09:24:36.115 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=2/16,p=0}@e005e9 IDLE
09:24:36.116 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=2/16,p=0} tryExecute STOP!
09:24:36.116 [qtp231500219-152] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=1/16,p=0}@16804601 offer STOP!
09:24:36.116 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=1/16,p=0}@16804601 task=STOP!
09:24:36.116 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.ReservedThreadExecutor - ReservedThreadExecutor@76b436fa{s=1/16,p=0}@16804601 Exited
09:24:36.116 [qtp231500219-215] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - ran ReservedThreadExecutor@76b436fa{s=1/16,p=0}@16804601 in QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=13<=200,i=1,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]
09:24:36.120 [qtp231500219-217] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - shrinking QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=13<=200,i=2,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]
09:24:36.120 [qtp231500219-217] DEBUG org.eclipse.jetty.util.thread.QueuedThreadPool - Thread[qtp231500219-217,5,main] exited for QueuedThreadPool[qtp231500219]@dcc69bb{STARTED,8<=12<=200,i=1,r=16,q=0}[ReservedThreadExecutor@76b436fa{s=1/16,p=0}]

Cassandra to Graphql Naming Resolution

Cassandra's table and column names support full UTF while graphql only supports

/^[_a-zA-Z][_a-zA-Z0-9]*$/.

The resolution should have the follow attributes:

Be deterministic with schema changes and schema refreshes

Keep production traffic intact on schema changes

Be clear to the user

A proposal is to:

1. Remove surrounding quotes2. Change all names to lower camel for field names, upper camel for table names3. Convert any characters out of graphql bounds to its hex encoded equivalent4. If it starts with a number, prefix it with an underscore

On conflicts:

The non-altered field name is preferred and the other field names are hidden*If there is no un-alterned field and there is a conflict, the last field is chosen alphabetically and the other fields are hidden

Additionally, tables or columns created through the graphql endpoint should carry out the action and return an error message along with the response informing the user of the limitation.

Examples:

🕷 => x1F577"123" => _123"Hello world! => helloWorld

/schema/keyspaces/{ks} does not show replication information (only name) if SimpleStrategy is used

As of today:

[tomek:~] $ http GET http://$STARGATE:8082/schemas/keyspaces/ks1 X-Cassandra-Token:$TOKEN
HTTP/1.1 200 OK
Content-Length: 23
Content-Type: application/json
Date: Mon, 14 Sep 2020 09:32:39 GMT
Vary: Accept-Encoding

{
    "data": {
        "name": "ks1"
    }
}

Looking at swagger it should show RF?

{
  "data": {
    "name": "killrvideo",
    "datacenters": [
      {
        "name": "dc1",
        "replicas": 3
      },
      {
        "name": "dc2",
        "replicas": 3
      }
    ]
  }
}

Make it possible to measure code coverage using both UT and IT

Rationale:It is a good metric to look for in terms of testing completeness.Also a good one if to look for before open sourcing the code.

Current situation:

very little UT majority of tests intestingmodule (integration testing)

running jacoco straight hits some issues due to duplicate classes

trying to collect fromtesting(that uses failsafe) I got coverage only for "test" classes not for product classes

Acceptance criteria:

make it possible to run some maven command that will include code coverage or include jacoco reporting in theverifytask

so that eventually it can be imported to Sonarqube or any other tool

code coverage from UT is collected

code coverage from IT (testing module) on product classes is collected

it is not required to have this coverage merged but it would be nice

Support union and interface types

For people who are using schema-first development, union and interface types are often required to express complex use cases. Unions and interface types are frequently used.

CCM command on mac issues

CCM start gives the following message:
Library/Python/2.7/site-packages/ccm-2.8.3-py2.7.egg/ccmlib/cluster_factory.py:21: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
/Library/Python/2.7/site-packages/ccm-2.8.3-py2.7.egg/ccmlib/node.py:128: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
[node1 ERROR] intx ThreadPriorityPolicy=42 is outside the allowed range [ 0 ... 1 ]
Improperly specified VM option 'ThreadPriorityPolicy=42'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Rest API: allow specifying client side CL

At the time of this ticket the CL is hardcoded as LOCAL_QUORUM for the REST service.

Some use cases need a higher CL than LOCAL_QUORUM (LOCAL_SERIAL and SERIAL come to mind as common ones when paired with LWT). More advanced users maybe fine with LOCAL_ONE.

I suggest a default CL in the server config and a per request override option. The names don't have to be the existing CLs this is a good opportunity to have something like Dynamo or CosmosDB style consistency levels using the CosmosDB example herehttps://docs.microsoft.com/en-us/azure/cosmos-db/consistency-levels

STRONG == LOCAL_SERIAL

CONSISTENT == LOCAL_QUORUM

EVENTUAL == LOCAL_ONE

Standardize unit testing framework

Right now we apparently have a mixture of JUnit 4 and 5 in test code. It would be nice to have only the JUnit 5 framework.

Review and improve error handling throughout modules

CI: Add a job to test starctl script

We should test that the script to start stargate does what is supposed to.

For example:

The following command should create a stargate node that uses the node1 as storage node.

ccm create test -n 1 -s -b
./starctl --cluster-name test --cluster-seed 127.0.0.1 --cluster-version 3.11 --listen 127.0.0.10 --simple-snitch

The following command should create a single stargate node that joins the ring

./starctl --cluster-name stargate --cluster-version 3.11 --listen 127.0.0.1 --simple-snitch --developer-mode

...

┆Issue is synchronized with this Asana task by Unito

Consistent logging

In Stargate many modules import logback-core, which is probably not ideal. It make log operation obscure in runtime since logging is not controlled in one place.

Most modules should probably import only slf4j. Then, we could have a separate OSGi bundle that would contain an isolated set of jars implementing slf4 and control logging in runtime.

Downstream project will have the option of providing their own log implementation bundles or simply overwrite the default logback.xml config.

testing pom.xml has hardcoded version

I think this line in testing/pom.xml

        <version>1.0-SNAPSHOT</version>

should be

        <version>${revision}</version>

Otherwise I got this error:

stargate $ mvn clean
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for io.stargate.it:testing:1.0-SNAPSHOT: Could not find artifact io.stargate:stargate:pom:1.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 7, column 13

C* nodes added to the cluster seen as peers by Stargate

When a node is added to a cluster where Stargate nodes are already running, that node is added to the peers list of all the Stargate nodes and will serve requests directly as coordinator:
[file id="cdb627cf-ff8d-48fd-82b0-97215ba34b2d"]

If Stargate nodes are then restarted, things go back to normal and only stargate nodes appear in the peers list of other Stargate nodes.

Improve error responses from RestApi

Go through errors returned to the users and add more details.

Use spotless for copyright headers

The Spotless plugin is helpful in making sure copyright headers are consistent across all files and it also helps with adding initial headers (viamvn spotless-apply).

Here's an example config (obviously we need to use Apache 2 header text):

<plugin>
        <groupId>com.diffplug.spotless</groupId>
        <artifactId>spotless-maven-plugin</artifactId>
        <version>${spotless.version}</version>
        <executions>
          <execution>
            <!-- run during verify phase -->
            <phase>verify</phase>
            <goals>
              <goal>check</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <formats>
            <format>
              <includes>
                <!-- Include all property files in "resource" folders under "src" -->
                <include>prod-endpoint/core/src/**/*.properties</include>
              </includes>
              <licenseHeader>
                <content>
#
# Copyright DataStax, Inc.
#
# Please see the included license file for details.
#
                </content>
                <!-- src file body until first occurrence of the delimiter regex will be interpreted as header section -->
                <delimiter>^$</delimiter>
              </licenseHeader>
            </format>
          </formats>
          <java>
            <includes>
              <include>prod-endpoint/**/src/**/java/**/*.java</include>
            </includes>
            <licenseHeader>
              <content>
/*
 * Copyright DataStax, Inc.
 *
 * Please see the included license file for details.
 */
${line.separator}
              </content>
            </licenseHeader>
          </java>
          <scala>
            <includes>
              <include>prod-endpoint/**/src/**/*.scala</include>
            </includes>
            <licenseHeader>
              <content>
/*
 * Copyright DataStax, Inc.
 *
 * Please see the included license file for details.
 */
${line.separator}
              </content>
            </licenseHeader>
          </scala>
        </configuration>
      </plugin>

Surface Indexes

Indexes should be treated as directives for schema evolution and migration purposes.

type Tweet 
@Index(name: "MyCustomIndex", type: StorageAttachedIndex, on: [ username ])
@PrimaryKey(field: [tweet])
{
  tweet: Text
  username: Text
}