GithubHelp home page GithubHelp logo

thomasnield / rxkotlin-jdbc Goto Github PK

View Code? Open in Web Editor NEW
28.0 4.0 4.0 74 KB

Fluent RxJava JDBC extension functions for Kotlin

License: Apache License 2.0

Kotlin 100.00%
rxjava rxjava-jdbc kotlin rxkotlin rx jdbc reactivex

rxkotlin-jdbc's Introduction

RxKotlin-JDBC

Fluent, concise, and easy-to-use extension functions targeting JDBC in the Kotlin language with RxJava 2.0.

This library is inspired by Dave Moten's RxJava-JDBC but seeks to be much more lightweight by leveraging Kotlin functions. This works with threadpool DataSource implementations such as HikariCP, but can also be used with vanilla JDBC Connections.

Extension functions like select(), insert(), and execute() will target both DataSource and JDBC Connection types. There is also support for safe blocking Sequence operations.

Binaries

Maven

<dependency>
  <groupId>org.nield</groupId>
  <artifactId>rxkotlin-jdbc</artifactId>
  <version>0.4.1</version>
</dependency>

Gradle

repositories {
    mavenCentral()
}
dependencies {
    compile 'org.nield:rxkotlin-jdbc:0.4.1'
}

Managing JDBC with Observables and Flowables

Using DataSources

When you use a DataSource, a Connection will automatically be pulled from the pool upon subscription and given back when onComplete is called.

val config = HikariConfig()
config.jdbcUrl = "jdbc:sqlite::memory:"
config.minimumIdle = 3
config.maximumPoolSize = 10

val ds = HikariDataSource(config)

//initialize

with(ds) {
    execute("CREATE TABLE USER (ID INTEGER PRIMARY KEY, USERNAME VARCHAR(30) NOT NULL, PASSWORD VARCHAR(30) NOT NULL)")
    execute("INSERT INTO USER (USERNAME,PASSWORD) VALUES (?,?)", "thomasnield", "password123")
    execute("INSERT INTO USER (USERNAME,PASSWORD) VALUES (?,?)", "bobmarshal","batman43")
}

// Retrieve all users
ds.select("SELECT * FROM USER")
        .toObservable { it.getInt("ID") to it.getString("USERNAME") }
        .subscribe(::println)


// Retrieve user with specific ID
ds.select("SELECT * FROM USER WHERE ID = :id")
        .parameter("id", 2)
        .toSingle { it.getInt("ID") to it.getString("USERNAME") }
        .subscribeBy(::println)

// Execute insert which return generated keys, and re-select the inserted record with that key
ds.insert("INSERT INTO USER (USERNAME, PASSWORD) VALUES (:username,:password)")
        .parameter("username","josephmarlon")
        .parameter("password","coffeesnob43")
        .toFlowable { it.getInt(1) }
        .flatMapSingle {
            conn.select("SELECT * FROM USER WHERE ID = :id")
                    .parameter("id", it)
                    .toSingle { "${it.getInt("ID")} ${it.getString("USERNAME")} ${it.getString("PASSWORD")}" }
        }
        .subscribe(::println)

// Run deletion

conn.execute("DELETE FROM USER WHERE ID = :id")
        .parameter("id",2)
        .toSingle()
        .subscribeBy(::println)

Using Connections

You can also use a standard Connection with these extension functions, and closing will not happen automatically so you can micromanage the life of that connection. It is also helpful to execute transactions against an individual connection.

val connection = DriverManager.getConnection("jdbc:sqlite::memory:")

connection.select("SELECT * FROM USER")
        .toObservable { it.getInt("ID") to it.getString("USERNAME") }
        .subscribe(::println)
        
connection.close()

Blocking Sequences

It can be convenient to work with JDBC fluently in a blocking manner, so you don't always have to return everything as an Observable or Flowable. While you should strive to not break the monad, sometimes it is easier to not have items returned in reactive fashion. This is especially the case when you have reactively built T objects, but you want to retrieve metadata U and V objects when T is constructed.

You could try to use RxJava's native blocking operators, but these can have undesired side effects of interruption exceptions especially when the blocking operation is happening under a parent reactive operation that can be disposed. Therefore, these Sequence parts of the API do not rely on RxJava at all and are implemented at the Iterable level. These may be candidate to be moved to a separate library (and be used as a dependency in this one) before rxkotlin-jdbc reaches a 1.0 release. This transition is not planned to be very breaking, other than perhaps a few import changes.

For instance, we can call toSequence() to iterate a ResultSet as a Sequence.

data class User(val id: Int, val userName: String, val password: String)

conn.select("SELECT * FROM USER").toSequence {
    User(it.getInt("ID"), it.getString("USERNAME"), it.getString("PASSWORD"))
}.forEach(::println)

The toSequence() actually returns a ResultSetSequence, which is important because if you use Sequence operators like take(), the iteration will need to manually be terminated early (completion events are not communicated like in RxJava). The returned ResultSetSequence should provide a handle to do this.

val conn = connectionFactory()

conn.select("SELECT * FROM USER").toSequence { it.getInt("ID") }.apply {

        take(1).forEach(::println)
        close() // must manually close for operators that terminate iteration early
}

Typically, if you simply want to return one item from a ResultSet, just call blockingFirst() or blockingFirstOrNull() which will handle the close() operation for you.

val conn = connectionFactory()

val id = conn.select("SELECT * FROM USER WHERE ID = 1").blockingFirst { it.getInt("ID") }

println(id)

Allowing Choice of Sequence, Observable, or Flowable

This library supports emitting T items built off a ResultSet in the form of:

  • Sequence<T>
  • Observable<T>
  • Flowable<T>
  • Single<T>
  • Maybe<T>

If you are building an API, it may be handy to allow the user of the API to choose the means in which to receive the results.

The toPipeline() function will allow mapping the ResultSet to T items, but defer to the API user how to receive the results.

data class User(val userName: String, val password: String)

fun getUsers() = conn.select("SELECT * FROM USER")
                .toPipeline {
                    User(it.getString("USERNAME"), it.getString("PASSWORD"))
                }

fun main(args: Array<String>) {

    getUsers().toFlowable().subscribe { println("Receiving $it via Flowable") }

    getUsers().toObservable().subscribe { println("Receiving $it via Observable") }

    getUsers().toSequence().forEach { println("Receiving $it via Sequence") }
}

Batching

Batch operations are now supported in RxKotlin-JDBC. This allows you to write a large amount of data to a database quickly using JDBC's batching interface.

Simply have an Observable<T>, Flowable<T>, Iterable<T>, or Sequence<T> of desired elements to INSERT, UPDATE, or some other action against your database, and pass it to the batchExecute() function on Connection or DataSource.

You will also need to specify the SQL template, the mapper to turn each T element into SQL parameters, and the batch size.

You can have the result come back as an Observable<T>, Flowable<T>, Sequence<T>, or a Compeletable.

val conn = connectionFactory()

class User(val username: String, val password: String)

val insertElements = Flowable.just(
        User("josephmarlon", "coffeesnob43"),
        User("samuelfoley","shiner67"),
        User("emilyearly","rabbit99"),
        User("johnlawrey", "shiner23"),
        User("tomstorm","coors44"),
        User("danpaxy", "texas22"),
        User("heathermorgan","squirrel22")
)

conn.batchExecute(
        sqlTemplate = "INSERT INTO USER (USERNAME, PASSWORD) VALUES (:username,:password)",
        elements = insertElements,
        batchSize = 3,
        parameterMapper = {
            parameter("username", it.username)
            parameter("password", it.password)
        }
).toFlowable().count().subscribeBy { println("Inserted $it records") }

Building Where Conditions Fluently

An experimental feature in RxKotlin-JDBC is an API-friendly builder for WHERE conditions, that may or may not use certain parameters to build WHERE conditions.

For instance, here is a function that will query a table with three possible parameters:

fun userOf(id: Int? = null, userName: String? = null, password: String? = null) =
        conn.select("SELECT * FROM USER")
                .whereOptional("ID", id)
                .whereOptional("USERNAME", userName)
                .whereOptional("PASSWORD", password)
                .toObservable(::User)

We can then query for any combination of these three parameters (including none of them).

userOf().subscribe { println(it) } // prints all users

userOf(userName = "thomasnield", password = "password123")
     .subscribe { println(it) }  // prints user with ID 1


userOf(id = 2).subscribe { println(it) } // prints user with ID 2

Instead of providing a simple field name, we can also provide an entire templated expression for more complex condtions.

conn.select("SELECT * FROM USER")
       .whereOptional("ID > ?", 1)

rxkotlin-jdbc's People

Contributors

thomasnield avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

rxkotlin-jdbc's Issues

Add asList() and asMap() operators

fun ResultSet.asList() =  (1..this.metaData.columnCount).asSequence().map {
    this.getObject(it)
}.toList()

fun ResultSet.asMap() =  (1..this.metaData.columnCount).asSequence().map {
    metaData.getColumnName(it) to getObject(it)
}.toMap()

Support named parameters

Example 1:

        conn.select("SELECT * FROM USER WHERE ID = :id AND FLAG = :flag")
                .setInt("id", 4)
                .setBoolean("flag", true)
                .toFlowable { it.getInt("ID") to it.getString("USERNAME") }
                .subscribe(::println)

Example 2:

        conn.select("SELECT * FROM USER WHERE ID = :id AND FLAG = :flag")
                .setParam("id", 4)
                .setParam("flag", true)
                .toFlowable { it.getInt("ID") to it.getString("USERNAME") }
                .subscribe(::println)

Example 3:

        conn.select("SELECT * FROM USER WHERE ID = :id AND FLAG = :flag")
                .setParams("id" to 4, "flag" to true)
                .toFlowable { it.getInt("ID") to it.getString("USERNAME") }
                .subscribe(::println)

Connection leak on execute (Hikari)

I'm using rxkotlin-jdbc and a Hikari data source in a TornadoFX project and found that a connection is leaking whenever I use fun DataSource.execute to delete objects from an underlying PostgreSQL database. I came across this by enabling logging with the JVM parameter -Dorg.slf4j.simpleLogger.log.com.zaxxer.hikari=debug and setting the hikari config's leakDetectionThreshold property to 60_000 ms, which showed the following output in my console:

[HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
[HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[HikariPool-1 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
[HikariPool-1 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for org.postgresql.jdbc.PgConnection@3fd29f40 on thread tornadofx-thread-2, stack trace follows
java.lang.Exception: Apparent connection leak detected
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at org.nield.rxkotlinjdbc.DatasourceKt$execute$1.invoke(datasource.kt:78)
	at org.nield.rxkotlinjdbc.DatasourceKt$execute$1.invoke(datasource.kt)
	at org.nield.rxkotlinjdbc.PreparedStatementBuilder.toPreparedStatement(PreparedStatementBuilder.kt:80)
	at org.nield.rxkotlinjdbc.UpdateOperation$toSingle$1.call(UpdateOperation.kt:38)
	at org.nield.rxkotlinjdbc.UpdateOperation$toSingle$1.call(UpdateOperation.kt:6)
	at io.reactivex.internal.operators.single.SingleDefer.subscribeActual(SingleDefer.java:36)
	at io.reactivex.Single.subscribe(Single.java:3394)
	at io.reactivex.Single.subscribe(Single.java:3380)
	at <<my tornadofx view class>>$deleteSelected$1.invoke(<<my tornadofx view class>>:157)
	at <<my tornadofx view class>>$deleteSelected$1.invoke(<<my tornadofx view class>>:17)
	at tornadofx.FXTask.call(Async.kt:459)
	at javafx.concurrent.Task$TaskCallable.call(Task.java:1423)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
o

The specific function in violation:

fun deleteRegistrations(registrations: List<Registration>, idexam: Int) = db.execute(
    "DELETE FROM registered WHERE idexam = ? AND code in (${registrations.joinToString { "?" }});").parameter(
    idexam).parameters(*registrations.map {
    it.code
}.toTypedArray()).toSingle()

jdk: 1.8.0_144
kotlin version: 1.3.10
rxkotlinfx maven version: 2.2.2
hikaricp maven version: 3.2.0

Not sure this is a bug of Hikari or rxkotlin-jdbc, but I found that swapping the execute to insert (and toSingle() to toSingle{true}) fixes the leak. Seems like the connection is not being closed and returned to the pool correctly?

Unexpected behavior of InsertOperation.parameters

I'm trying to perform a batch insert into a PostgreSQL database. I'm trying to use a multi-insert statement (search for multirow values syntax at https://www.postgresql.org/docs/11/sql-insert.html) rather than a batch of single-insert statements. To this end, I'm building a template string with a set of parameter placeholders (question marks) and using the parameters method to set their values.

The code (call test and pass a connection to a PostgreSQL database):

data class Person(val firstName: String, val lastName: String)

fun upsertTemplate(n: Int) = "INSERT INTO person (firstname, lastname) VALUES ${(1..n).joinToString { "(?,?)" }}"

fun upsert(persons: List<Person>, connection: Connection) {
    val blockingGet = connection.insert(upsertTemplate(persons.size))
        .parameters(persons.flatMap { listOf(it.firstName, it.lastName) })
        .toObservable { 1 }
        .toList()
        .blockingGet()
}

fun test(connection: Connection) {
    upsert(listOf(Person("john", "doe"), Person("jane", "doe")), connection)
}

When executing this, I get an exception:

org.postgresql.util.PSQLException: No value specified for parameter 1.

If instead I insert the parameters one at a time (looping over the parameters list), everything is working as expected:

fun upsert(persons: List<Person>, connection: Connection) {
val insertOperation = connection.insert(upsertTemplate(persons.size))
    persons.flatMap { listOf(it.firstName, it.lastName) }
        .forEach {
            insertOperation.parameter(it)
        }
    val blockingGet = insertOperation
        .toObservable { 1 }
        .toList()
        .blockingGet()
}

Am I doing something wrong here? I don't want to use batchExecute because I don't want to perform a lot of single insert statements in 1 transaction, I want to use PostgreSQL's multirow insert.

Issues with nameless parameters

This function inside PreparedStatementBuilder:

    fun parameters(vararg parameters: Any?) {
        furtherOps += { it.processParameters(parameters) }
    }

Needs to be changed to:

    fun parameters(vararg parameters: Any?) {
        parameters.forEach { parameter(it) }
    }

That would then delegate to this function:

    fun parameter(value: Any?) {
        furtherOps += { it.processParameter(namelessParameterIndex.getAndIncrement(), value) }
    }

which handles the auto-incrementing of indices for nameless parameters.

Add toPipeline()

Sometimes it can be helpful for API's to allow the client to choose consuming a query as a Flowable, an Observable, or a Sequence.

It would also be nice if the (ResultSet) -> T operation was already set regardless of the return pipeline type.

Here is the solution:

class Pipeline<T: Any>(val selectOperation: SelectOperation,
                  val mapper: (ResultSet) -> T
) {
    fun toObservable(): Observable<T> = selectOperation.toObservable(mapper)
    fun toFlowable(): Flowable<T> = selectOperation.toFlowable(mapper)
    fun toSequence(): ResultSetSequence<T> = selectOperation.toSequence(mapper)
    fun blockingFirst() = selectOperation.blockingFirst(mapper)
    fun blockingFirstOrNull() = selectOperation.blockingFirstOrNull(mapper)
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.