eclipse-vertx / vertx-sql-client Goto Github PK

View Code? Open in Web Editor NEW

870.0 44.0 194.0 11.47 MB

High performance reactive SQL Client written in Java

License: Apache License 2.0

Java 98.82% Shell 0.05% Dockerfile 0.01% PLSQL 0.17% PLpgSQL 0.59% TSQL 0.37%

vertx postgres pg async netty performance scalability mysql reactive non-blocking

vertx-sql-client's People

Contributors

Stargazers

Watchers

Forkers

franz1981 billyyccc cazacugmihai darkredz trho gunnarmorling sangkyunyoon phisgr fossabot orelgenya pratikpparikh brosenfi emrul cxz davidzzzzz gimlet2 wizzardo sumitsrv marcust alejandrotamayo oshai hanumanacholia saminfante tsabirgaliev jklingsporn rasonyang oci-pronghorn arrowwood49 performance-folder ybonnel yonigibbs chakra-coder aaloise uuppala lanceon foxgem ixtli mp911de wiltonlazary rgielen danielamorais huntj88 mehcode xp13910818313 vietj mystdeim ttgoding slinkydeveloper misselvexu researchmore larousso u2name6 indiketa jeoffreylim benbenw cocktail18 rodrigosaladoanaya ryouginevermore yubobo iamfive aguibert globegitter luishiga lovelyefyyucui huangjinteng tsegismont wolf-21 chao-huang zhaoshiling1017 adedayoominiyi mswatosh xieguiqing jrversion hongzhe nml5136 gjwatts daltrogama sanne zenios julianladisch fromage anguisa guos tao1445 barreiro johnaohara davided hoaronal-project thorkarlsson zhangxinyutop folio-org zhbotong mu-l nandakishoremutyala adamdickmeiss emadalblueshi scaleplandev chno9133 aixingling pjgg

vertx-sql-client's Issues

Enumerated (enum) type support

Column checker for removing boiler plate test code

currently the data type test have lot of boiler plate code

Rename PgConnectionPool to PgPool

Minimize flush

we should investigate to minimizing flushes:

a command might write several netty messages and we flush on each
when pipelining, we could dequeue the pending queue by batches instead of one at a time

PgUpdate support

Support updates via prepared statements with an API similar to PgQuery:

PgPreparedStatement ps = client.prepare("INSERT INTO USERS (id, name) VALUES ($1, $2));
PgUpdate update = ps.update("julien", "Julien Viet");
update.execute(ar -> {
  // Handle result
});

// Or fluently
ps.update("julien", "Julien Viet").execute(ar -> {
  // Handle result
});

Does Reactive-pg-client support MySQL in Vertx?

Hello, I find it is not convenient to use vertx-mysql-postgresql-client in vertx，and it don't support batch operation. I find Reactive-pg-client and it Ranked #1 in the TechEmpower Benchmark Round 15 Single query benchmark.
I just want to know, Does Reactive-pg-client support MySQL in Vertx? Or is there any plan to suport MySQL in Vertx?

Array support

function inside batch update?

Hi,

Can I use functions inside sql of conn.preparedBatch(sql, batch, handler)? I tried something like:

INSERT INTO table_x (x, y) VALUES ($1, some_func($2));

, which returns could not determine data type of parameter $2.

Numeric array support

Rename project to reactive-pg-client

How to create a table using a PgClient

How do I execute a statement like this CREATE TABLE IF NOT EXISTS movies (id SERIAL, name varchar(100), release varchar(10)) using a PgClient?

Binary json and jsonb support

Pool improvements

connection timeout
idle timeout

Numeric as Unaltered BigDecimal

I am evaluating this library as an alternative to a couple of other libraries that have stalled. I am reading: https://github.com/vietj/reactive-pg-client/blob/09c07499bbe2925ade22b3b23a46552d08bcd3ea/src/main/java/com/julienviet/pgclient/impl/codec/DataType.java#L188-L203 and I see that it turns it to a double or big integer which seems very unnatural. As noted there I lose precision. Also, to work with it properly on the client side, I have to test for BigInteger or Double to change it back to a common type even though it is a common type on the postgres side.

Is there any chance to get BigDecimal support instead or is there anything I am misreading? Or can I use a custom data type? I hope the project is early enough that these kinds of changes aren't too harmful.

Implement close hook

SSL support

[API improvement] Types of params of PreparedStatement in Query or Update

Status

Currently query or update in PgOperations use List<Object> to hold multiple params, API is something like this void query(String sql, List<Object> params, Handler<AsyncResult<ResultSet>> handler);
or void update(String sql, List<Object> params, Handler<AsyncResult<UpdateResult>> handler);

Motivation

As it's defined in vertx-sql-common, SQLOperations use JsonArray to hold mutiple params like SQLOperations queryWithParams(String sql, JsonArray params, Handler<AsyncResult<ResultSet>> resultHandler);
or SQLOperations updateWithParams(String sql, JsonArray params, Handler<AsyncResult<UpdateResult>> resultHandler);
It's better to keep consistency, is it right?

Apart from this, List<Object> is also not supported by Codegen when passing as input parameters, so the method has to be ignored by @GenIgnore. When using Rxi-fied API, there is no methods to take multiple params as one collection, users have to maintain a state of how many params they use and then choose which method to use, this is not convenient when facing dynamic SQLs.

Therefore is there any possibility to change this?

Complete data type support

Numeric, geometric types, etc...

Connection in the pool closed by the database won't be removed from the available connections

UUID data type feature

ExtendedQueryResultHandler must check the async result is succeeded before unboxing its boolean value

I'm firing the same insert query two times and expect to get a database exception, but I get a NPE instead. It seems to be a autoboxing issue: when there is an exception, ExtendedQueryResultHandler.handle is called and res returns null which cannot be casted to primitive boolean.

Version: 0.5.2
Stacktrace:
17:58:42.481 [vert.x-eventloop-thread-0] ERROR io.vertx.core.impl.ContextImpl - Unhandled exception java.lang.NullPointerException: null at com.julienviet.pgclient.impl.ExtendedQueryResultHandler.handle(ExtendedQueryResultHandler.java:45) ~[reactive-pg-client-0.5.2.jar:na] at com.julienviet.pgclient.impl.ExtendedQueryResultHandler.handle(ExtendedQueryResultHandler.java:24) ~[reactive-pg-client-0.5.2.jar:na] at com.julienviet.pgclient.impl.CommandBase.handleMessage(CommandBase.java:44) ~[reactive-pg-client-0.5.2.jar:na] at com.julienviet.pgclient.impl.QueryCommandBase.handleMessage(QueryCommandBase.java:51) ~[reactive-pg-client-0.5.2.jar:na] at com.julienviet.pgclient.impl.ExtendedQueryCommandBase.handleMessage(ExtendedQueryCommandBase.java:68) ~[reactive-pg-client-0.5.2.jar:na] at com.julienviet.pgclient.impl.ExtendedQueryCommand.handleMessage(ExtendedQueryCommand.java:31) ~[reactive-pg-client-0.5.2.jar:na] at com.julienviet.pgclient.impl.SocketConnection.handleMessage(SocketConnection.java:233) ~[reactive-pg-client-0.5.2.jar:na]

Binary Decode/Encode for character types

Behaviour of exhausted pool?

Hi,
I'm getting an issue when using the connection pool (that is, without manually managing my own connection). The issue is that if more than 10 concurrent requests are excited against a pool that only has 10 connections, there seems to be a deadlock condition and nothing ever happens.

I have updated my code from a previous test in order to demonstrate this. The variable " iterationCount)" can be used to test the condition. To make the test work, change it to iteractionCount = 5.

Please checkout project and issue a "grade run" to replicate

https://github.com/willks/pgtest

Large object support

Upgrade to Vert.x 3.5.1

Handle NUMERIC type with a custom Number

motivation: currently NUMERIC type is mapped to BigDecimal which does not allow a full representability of the type (such as ∞).
changes: provide a specific subclass of Number (perhaps named PgNumeric ?) that wraps the value as a BigDecimal when possible otherwise indicates which specific value it is. This can then be converted by the Row class to other values or can be retrieved as is. This mapping shall be bidirectional.

Default text encoding assumes the ByteBuf is large enough for encoding the value

The following code gives an error:

        List<Tuple> batch = new ArrayList<>();
        batch.add(Tuple.of(79991, "batch one"));
        batch.add(Tuple.of(79992, "batch two"));
        batch.add(Tuple.of(79993, "batch three"));
        batch.add(Tuple.of(79994, "batch four"));

        conn.preparedBatch("SELECT $1::integer, $2::varchar(2048)", batch, event -> {
            if (event.succeeded()) {
                logger.info("Stored new batch of commands");
            } else {
                logger.info("Error storing batch of commands", event.cause());
            }
            async.complete();
        });

But with the following change (note added spaces in sql) the error disappears

        List<Tuple> batch = new ArrayList<>();
        batch.add(Tuple.of(79991, "batch one"));
        batch.add(Tuple.of(79992, "batch two"));
        batch.add(Tuple.of(79993, "batch three"));
        batch.add(Tuple.of(79994, "batch four"));

        conn.preparedBatch("SELECT $1::integer, $2::varchar(2048)                                                  ", batch, event -> {
            if (event.succeeded()) {
                logger.info("Stored new batch of commands");
            } else {
                logger.info("Error storing batch of commands", event.cause());
            }
            async.complete();
        });

The error that happens is

SEVERE: Unhandled exception
java.lang.IndexOutOfBoundsException: index: 253, length: 30 (expected: range(0, 256))
	at io.netty.buffer.AbstractByteBuf.checkIndex0(AbstractByteBuf.java:1360)
	at io.netty.buffer.AbstractByteBuf.checkIndex(AbstractByteBuf.java:1355)
	at io.netty.buffer.AbstractByteBuf.setCharSequence0(AbstractByteBuf.java:682)
	at io.netty.buffer.AbstractByteBuf.setCharSequence(AbstractByteBuf.java:672)
	at com.julienviet.pgclient.impl.codec.DataType.encodeText(DataType.java:655)
	at com.julienviet.pgclient.impl.codec.DataType$9.encodeBinary(DataType.java:244)
	at com.julienviet.pgclient.impl.codec.DataType$9.encodeBinary(DataType.java:234)
	at com.julienviet.pgclient.impl.codec.encoder.message.Bind.encode(Bind.java:154)
	at com.julienviet.pgclient.impl.codec.encoder.message.Bind.encode(Bind.java:167)
	at com.julienviet.pgclient.impl.SocketConnection.checkPending(SocketConnection.java:210)
	at com.julienviet.pgclient.impl.SocketConnection.bilto(SocketConnection.java:191)
	at com.julienviet.pgclient.impl.CommandBase.foo(CommandBase.java:54)
	at com.julienviet.pgclient.impl.SocketConnection.schedule(SocketConnection.java:181)
	at com.julienviet.pgclient.impl.ConnectionPool$PooledConnection.schedule(ConnectionPool.java:77)
	at com.julienviet.pgclient.impl.PgPoolImpl$1.lambda$onSuccess$0(PgPoolImpl.java:73)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:748)

Question: Example on storing JSONB column with prepared query

Hi,
I've been trying to store an embedded jsonb object, but column is always null and console prints
"Data type 3802 does not support binary encoding". I've tried saving as a "JsonObject" type - and the string representation of the JSON. Is there an example on how to do this that I missed? I searched through tests and examples (but probably not diligently enough!)

Create factory method for com.julienviet.reactivex.pgclient.Tuple that takes varargs or a Collection

Hi,
I was using a statement like this Tuple tuple = Tuple.of(bindValues.toArray()); (with bindValues being an ArrayList) which works fine for the non-rx version. When switching to rxified version of Tuple my code fails in that regard, that Tuple now consists of 1 entry (which is an array) instead of n entries.
I'm doing this workaround for now:

Tuple tuple = Tuple.tuple();
bindValues.forEach(tuple::addValue);

Introduce JSON wrapper type

motivation: we need to introduce a specific wrapper type for JSON to handle correctly null json values and distinguish them from SQL NULL

Provide one-shot commands

PgPool hold one shot version of the API that avoids the acquire/execute/release dance and just borrow one connection from the pool.

Provide way to access tuple by column name

Configure from env variables

Provide an option for building the options from the env like libpq : https://www.postgresql.org/docs/9.1/static/libpq-envars.html

Pub-sub support

Rework pom.xml to have javadoc jar installed on deploy

Remove DatatypeConverter usage in DataType

this prevents using the client in Java 9

Use reactiverse package and groupId

Connection acquisition incorrectly assumes that there will be a waiter in the pool

Hi,
I keep getting this in my logs when running concurrent tests. It uses 5 concurrent users - and the exception observed is this with the ReactiveX flavour:

SEVERE: Unhandled exception
java.lang.NullPointerException
	at com.julienviet.pgclient.impl.ConnectionPool.lambda$check$1(ConnectionPool.java:165)
	at com.julienviet.pgclient.impl.ConnectionPool.doAcq(ConnectionPool.java:139)
	at com.julienviet.pgclient.impl.ConnectionPool.lambda$doAcq$0(ConnectionPool.java:150)
	at com.julienviet.pgclient.impl.InitCommand.handleMessage(InitCommand.java:93)
	at com.julienviet.pgclient.impl.SocketConnection.handleMessage(SocketConnection.java:233)
	at io.vertx.core.net.impl.NetSocketImpl.handleMessageReceived(NetSocketImpl.java:351)
	at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:242)
	at io.vertx.core.net.impl.NetClientImpl$1.handleMessage(NetClientImpl.java:239)
	at io.vertx.core.net.impl.VertxHandler.lambda$channelRead$1(VertxHandler.java:146)
	at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:337)
	at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:195)
	at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:144)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:844)

My pool settings are:

        PgPoolOptions options = new PgPoolOptions()
                .setPort(5432)
                .setHost("vm")
                .setDatabase("xxxxx")
                .setUsername("yyyyy")
                .setPassword("zzzzz")
                .setConnectTimeout(100000)
                .setIdleTimeout(100000)
                .setCachePreparedStatements(true)
                .setLogActivity(true)
                .setReconnectAttempts(10)
                .setTcpFastOpen(true)
                .setMaxSize(25);

All queries issued return a Single, and I'm using the query code like this (rxPreparedQuery):

    public Single<T> get(UUID entityId, Identity identity) {
        Tuple binds = Tuple.of(entityId, identity.getIdentityId());
        return db.rxPreparedQuery("select * from " + typeParameterClass.getSimpleName().toLowerCase() +" where id=$1 and identityid=$2", binds).map(rowPgResult -> {
            if (rowPgResult.size() == 0) {
                return empty();
            }
            Row next1 = rowPgResult.iterator().next();
            return mapRow(next1);
        });
    };

The issue is intermittent, but as I remove my tests concurrency, the error appears less. Running the code more times seems to yield less errors. With a single user, on repeated tests the error will no longer appear.

Seems that this code gets triggered when I see it:

  public void handleMessage(InboundMessage msg) {
    if (msg.getClass() == ReadyForQuery.class) {
      ReadyForQuery readyForQuery = (ReadyForQuery) msg;
      completionHandler.handle(null);
      if (failure != null) {
        handler.handle(CommandResponse.failure(failure, readyForQuery.txStatus()));
      } else {
        handler.handle(CommandResponse.success(result, readyForQuery.txStatus()));
      }
    } else {
      System.out.println(getClass().getSimpleName() + " should handle message " + msg);
    }
  }

That is

        handler.handle(CommandResponse.failure(failure, readyForQuery.txStatus()));

has the status: txStatus=IDLE

Not sure if related - or that if the error is because the PreparedStatement doesn't exist? I can't figure it out yet.

Any clues?

Add Row-support for Short

According to the README, Short as a datatype is supported, but I can't find such getters/setters in Row or Tuple.

Bytea binary support

Ensure Row by name getters throws NPE for a null column name

Handling JSON/JSONB scalar types

Currently only Json Object and Json Array are handled properly. Scalar types are missing.

Replace with usage of ByteBuf#skipBytes(n)

which is more efficient for skipping bytes, we use sometime the pattern buff.readerIndex(buff.readerIndex() + n)

Request cancellation support

Tuple with more than 6 argument?

Hi,
Did not know if it was the best place to ask a question, but I could not find somewhere else. So I'm building a fully reactive app using gRPC, and this driver was of interest to me. The syntax is small and concise.

The issue I have is that I'm doing an insert, and the insert has more than 6 tuples. I want to use bindings as I don't like encoding values in the string:

Code:

public Single<Identity> create(Identity identity) {
    long timestamp = System.currentTimeMillis();
    identity.setCreationTimestamp(timestamp);
    identity.setModificationTimestamp(timestamp);
    identity.setIdentityId(UUID.randomUUID());
    Single<PgResult<Row>> insert = db.rxPreparedQuery(
            "insert into principal(id, identityId, name, email, creationTimestamp, modificationTimestamp, active) " +
                    "values ($1, $2, $3, $4, $5, $6, $7) returning *" ,
            Tuple.of(identity.getId(),
                    identity.getIdentityId().toString(),
            identity.getName(),
            identity.getEmail(),
            identity.getCreationTimestamp(),
            identity.getModificationTimestamp()));
    return insert.map(new Function<PgResult<Row>, Identity>() {
        @Override
        public Identity apply(PgResult<Row> rowPgResult) throws Exception {
            // We only expect one result
            PgResult<Row> next = rowPgResult.next();
            Row next1 = next.iterator().next();
            return mapRow(next1);
        }
    });
}

However, I run out of tuple arguments for the 7th parameter! Any suggestions?

Thanks

Ensure Row by name getters return null when the column does not exists

Missing RxJava 1 and 2 javadocs

Hello,

I am keen to try this as a replacement for our postgres driver with com.impossible for pg_notify support. RxJava is said to be supported, but I do not see anywhere Rx methods or a wrapper Rx library like RxJava for Vert.x. What am I missing?

Thank you.

Configure from URI

e.g : postgresql://dbuser:[email protected]:3211/mydb

Remove PgConnection preparedAnd* operations

Currently the client provides preparedAnd* operations:

connection.prepareAndQuery("SELECT * FROM USERS WHERE id=$1", "julien", ar -> {});

we should remove them, users can do instead:

connection.prepare("SELECT * FROM USERS WHERE id=$1").query("julien").execute(ar -> {});

eclipse-vertx / vertx-sql-client Goto Github PK

vertx-sql-client's People

Contributors

Stargazers

Watchers

Forkers

vertx-sql-client's Issues

Status

Motivation

Recommend Projects

Recommend Topics

Recommend Org

Jobs