mysql match operator about ktorm

[schmeic]

Looks like this is not currently part of the MysqlDialect, and I'm not sure what the best approach would be to handle both the "match" keyword (for multiple columns) and the "against" keyword (with modes). Any suggestions?

[vincentlauvlwj]

Thanks for your feedback, we will add this feature to the next release.

If you really need it currently, please do the following steps.

1. Create a class that represents match ... against ... expressions extending from ScalarExpression. An enum class MatchAgainstModifier that represents the modes is also needed.

data class MatchAgainstExpression(
    val matchColumns: List<ColumnExpression<*>>,
    val searchString: String,
    val modifier: MatchAgainstModifier? = null,
    override val sqlType: SqlType<Boolean> = BooleanSqlType,
    override val isLeafNode: Boolean = false
) : ScalarExpression<Boolean>()

enum class MatchAgainstModifier(private val value: String) {
    IN_NATURAL_LANGUAGE_MODE("in natural language mode"),
    IN_NATURAL_LANGUAGE_MODE_WITH_QUERY_EXPANSION("in natural language mode with query expansion"),
    IN_BOOLEAN_MODE("in boolean mode"),
    WITH_QUERY_EXPANSION("with query expansion");

    override fun toString(): String {
        return value
    }
}

2. Add DSL functions that can help us creating MatchAgainstExpression instances conveniently.

class MatchColumns(columns: List<ColumnExpression<*>>) : List<ColumnExpression<*>> by columns

fun match(vararg columns: Column<*>): MatchColumns {
    return MatchColumns(columns.map { it.asExpression() })
}

fun MatchColumns.against(searchString: String, modifier: MatchAgainstModifier? = null): MatchAgainstExpression {
    return MatchAgainstExpression(this, searchString, modifier)
}

3. Add a subclass of SqlFormatter, formatting MatchAgainstExpressions to SQL strings.

class CustomFormatter(database: Database, beautifySql: Boolean, indentSize: Int)
    : SqlFormatter(database, beautifySql, indentSize) {

    override fun visitUnknown(expr: SqlExpression): SqlExpression {
        if (expr is MatchAgainstExpression) {
            return visitMatchAgainst(expr)
        } else {
            return super.visitUnknown(expr)
        }
    }

    private fun visitMatchAgainst(expr: MatchAgainstExpression): MatchAgainstExpression {
        write("match (")
        visitExpressionList(expr.matchColumns)
        removeLastBlank()
        write(") against (?")
        _parameters += ArgumentExpression(expr.searchString, VarcharSqlType)
        expr.modifier?.let { write(" $it") }
        write(")")
        return expr
    }
}

4. Finally, create your custom dialect and register it to Ktorm while creating Database instance.

object CustomDialect : SqlDialect {
    override fun createSqlFormatter(database: Database, beautifySql: Boolean, indentSize: Int): SqlFormatter {
        return CustomFormatter(database, beautifySql, indentSize)
    }
}

val db = Database.connect(
    url = "jdbc:mysql://127.0.0.1:3306/ktorm",
    driver = "com.mysql.jdbc.Driver",
    user = "root",
    dialect = CustomDialect,
    logger = ConsoleLogger(threshold = LogLevel.TRACE)
)

Now you can use the match ... against ... feature, usage:

val employees = Employees.findList { 
    match(it.name, it.job).against("vince", MatchAgainstModifier.IN_NATURAL_LANGUAGE_MODE) 
}

Generated SQL:

select ... 
from t_employee 
left join t_department _ref0 on t_employee.department_id = _ref0.id 
where match (t_employee.name, t_employee.job) against (? in natural language mode)

I hope that would be useful to you. Enjoy Ktorm.

[schmeic]

Wow, awesome! Thanks for the quick reply. I've been using Exposed for awhile, and just found ktorm - my initial impression is that the code is a lot cleaner and simpler, and the documentation is way better. This match operator was the one thing holding me up from switching my app over to ktorm.

One minor suggestion - what do you think about just naming the enum "MatchMode", and then you probably don't need to IN or MODE, so like this:

val employees = Employees.findList { 
    match(it.name, it.job).against("vince", MatchMode.NATURAL_LANGUAGE) 
}

[schmeic]

Also, a bit off topic, but is there a slack channel (or something similar) for ktorm? Just for discussion of things that aren't necessarily issues for github.

For example, this functionality just released today by the Exposed team seems like an easy, concise way to make custom functions, maybe ktorm can consider something similar:
https://github.com/JetBrains/Exposed/wiki/Functions#custom-functions

[vincentlauvlwj]

Well, I named the enum class as MatchAgainstModifier just following the names in MySQL's reference manual: https://dev.mysql.com/doc/refman/5.5/en/fulltext-search.html

To make the name shorter, maybe MatchModifier is better.

[vincentlauvlwj]

match ... against ... is not just about custom functions, that's why we need to custom expression types and SqlFormatter here.

For simple SQL functions such as replace, we just need to add a simple extension function:

fun ColumnDeclaring<String>.replace(oldValue: String, newValue: String): FunctionExpression<String> {
    // replace(this, oldValue, newValue)
    return FunctionExpression(
        functionName = "replace",
        arguments = listOf(
            this.asExpression(),
            ArgumentExpression(oldValue, VarcharSqlType),
            ArgumentExpression(newValue, VarcharSqlType)
        ),
        sqlType = VarcharSqlType
    )
}

Usage:

val names = Employees.asSequence().mapColumns { it.name.replace("vince", "VINCE") }

Generated SQL:

select replace(t_employee.name, ?, ?)
from t_employee 
left join t_department _ref0 on t_employee.department_id = _ref0.id 

I think it's much better than Exposed's custom functions.

[schmeic]

I guess to really match the MySql docs, the best term would be SearchModifier. And you're right, it's best to make it as close as possible to the official docs. Either way, it's way better than what I had to do in Exposed:

val naturalLanguageMode = object : FunctionProvider.MatchMode {
        override fun mode() = "IN NATURAL LANGUAGE MODE"
}

[schmeic]

Thanks for sharing the code to implement the replace function. I agree, it's better than Exposed - I don't know why Exposed would bind these functions to a particular table column, rather than have it available to any string column.

What I do like about the Exposed custom function syntax is that it is pretty concise, but I suppose I can write my own wrapper. Something like:

createStringFunction(functionName: String, vararg args: ArgumentExpression)

[schmeic]

Sorry, one more thing - rather than creating a CustomDialect as in the example above, what I'd really like to to is extend the MysqlDialect. How would I do that? The MySqlFormatter is a private class so I don't think I can extend it.

[vincentlauvlwj]

It's a design mistake. I'm planing to make it public and open in the next release.

As a temporary solution, you can copy the code from MySqlDialect to your CustomDialect.

[schmeic]

Cool, thanks - I can wait for the next release. I really appreciate all your help. Btw, what do you think about trying to get a channel on the kotlinlang slack site?

[vincentlauvlwj]

Sorry, I don't use slack very much. I think GitHub issues are not bad for discussions.

[vincentlauvlwj]

I just released a new version v2.4 hours ago: https://github.com/vincentlauvlwj/Ktorm/releases/tag/v2.4

The new version supports the match ... against ... syntax just in the way we discussed, and MySqlFormatter is public and open now.