can not parsed quoted string about clickhouse4j

[Tasselmi]

Array(Uint32) like [0,12191]   is ok
but Array(String) like  ['071','122'] is not ok

[Tasselmi]

type: Array(UInt32), parsed text: "[0,12191]"
name: event_id,         type: Array(String), parsed text: <EMPTY>ERROR

[doom369]

@Tasselmi could you please provide minimal sql statement example?

[Tasselmi]

@Tasselmi could you please provide minimal sql statement example?

I use spark's default df.write.format("jdbc") method.

now I'm trying to maintain a connection pool by myself.
If also does not work, I will give more details.
Thanks

[doom369]

@Tasselmi ok. I'll try to reproduce by myself for now.

[doom369]

@Tasselmi pushed additional test - it works just fine. We need more info. Minimal reproducer would be nice.

[Tasselmi]

@Tasselmi pushed additional test - it works just fine. We need more info. Minimal reproducer would be nice.

CREATE TABLE test.test
(
    `age` UInt8,
    `name` Array(String)
)
ENGINE = Log

$ spark-shell --num-executors 1 --executor-memory 4g --executor-cores 1  --jars /home/me/jars/clickhouse4j-1.1.1.jar

scala> :paste
// Entering paste mode (ctrl-D to finish)

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{Row, SaveMode, SparkSession}
import org.apache.spark.storage.StorageLevel

// Exiting paste mode, now interpreting.



scala> val df = Seq((18, "liang"), (18, "fan")).toDF("age", "name")
df: org.apache.spark.sql.DataFrame = [age: int, name: string]

scala> df.show
+---+-----+
|age| name|
+---+-----+
| 18|liang|
| 18|  fan|
+---+-----+

scala> val dfa = df.groupBy("age").agg(collect_list("name").as("name_list"))
dfa: org.apache.spark.sql.DataFrame = [age: int, name_list: array<string>]

scala> dfa.show
+---+------------+
|age|   name_list|
+---+------------+
| 18|[liang, fan]|
+---+------------+

scala> :paste
// Entering paste mode (ctrl-D to finish)

  val rdd = dfa.rdd.map { row =>
      val age = row.getAs[Int]("age")
      val name = row.getAs[Seq[String]]("name_list")
      Row(age, name.map(e => "\'" + e + "\'").mkString("[", ",", "]"))
  }
  val schema = StructType(
    Seq(
      StructField("age", IntegerType),
      StructField("name", StringType)
    )
  )
  val dfb = spark.createDataFrame(rdd, schema)
  dfb.show()

// Exiting paste mode, now interpreting.

+---+---------------+
|age|           name|
+---+---------------+
| 18|['liang','fan']|
+---+---------------+


scala> :paste

  dfb.write
    .mode(SaveMode.Append)
    .format("jdbc")
    .option("url", dburl)
    .option("driver", driverClass4j)
    .option("dbtable", "test")
    .option("user", userMe)
    .option("password", passwordMe)
    .save()

ClickHouse exception, code: 26, host: null, port: 0; Code: 26, e.displayText() = DB::Exception: Cannot parse quoted string: expected opening quote: (at row 1)

Row 1:
Column 0,   name: age,  type: UInt8,         parsed text: "18"
Column 1,   name: name, type: Array(String), parsed text: <EMPTY>ERROR

 (version 19.16.2.2 (official build))

but just look at the printing:

+---+---------------+
|age|           name|
+---+---------------+
| 18|['liang','fan']|
+---+---------------+

the format is right....

[Tasselmi]

I use connectionPool to insert, and the error is the same.

This makes me puzzled a whole day.

looking forward your help.

[doom369]

@Tasselmi any way to view the generated sql?

[Tasselmi]

@Tasselmi any way to view the generated sql?

ok let me figure out.

[Tasselmi]

@Tasselmi any way to view the generated sql?

sorry I can not to get the generated sql.
But here are some clues.
money and name are string type in spark.
money is Array(Int) and name is Array(String) in clickhouse.

money works fine, but name fails.

import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.{Row, SaveMode, SparkSession}
import org.apache.spark.storage.StorageLevel

val dburl = "jdbc:clickhouse://localhost:8123/test"
val driverClass4j = "cc.blynk.clickhouse.ClickHouseDriver"

val df = Seq((18, 1000, "liang"), (18, 2000, "fan")).toDF("age", "money", "name")

val dfa = df.groupBy("age").agg(
    collect_list("money").as("money_list"),
    collect_list("name").as("name_list")
)


val rdd = dfa.rdd.map { row =>
    val age = row.getAs[Int]("age")
    val money = row.getAs[Seq[Int]]("money_list")
    val name = row.getAs[Seq[String]]("name_list")
    Row(age, 
        money.mkString("[", ",", "]"),
        name.map(e => "\'" + e + "\'").mkString("[", ",", "]")
    )
}
val schema = StructType(
    Seq(
      StructField("age", IntegerType),
      StructField("money", StringType),
      StructField("name", StringType)
    )
)
val dfb = spark.createDataFrame(rdd, schema)

scala>  dfb.show

+---+-----------+-------------+                                                 
|age|      money|         name|
+---+-----------+-------------+
| 18|[1000,2000]|'liang','fan'|
+---+-----------+-------------+

dfb.write
  .mode(SaveMode.Append)
  .format("jdbc")
  .option("url", dburl)
  .option("driver", driverClass4j)
  .option("dbtable", "test")
  .option("user", "default")
  .option("password", "")
  .save()


Caused by: java.lang.Throwable: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected [ before: \\\'liang\\\',\\\'fan\\\'\n: (at row 1)

Row 1:
Column 0,   name: age,   type: UInt8,         parsed text: "18"
Column 1,   name: money, type: Array(UInt16), parsed text: "[1000,2000]"
Column 2,   name: name,  type: Array(String), parsed text: <EMPTY>ERROR

 (version 19.16.2.2 (official build))

  at cc.blynk.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:56)
  ... 21 more

scala> dfb.show
+---+-----------+---------------+                                               
|age|      money|           name|
+---+-----------+---------------+
| 18|[1000,2000]|['liang','fan']|
+---+-----------+---------------+

dfb.write
  .mode(SaveMode.Append)
  .format("jdbc")
  .option("url", dburl)
  .option("driver", driverClass4j)
  .option("dbtable", "test")
  .option("user", "default")
  .option("password", "")
  .save()

Caused by: java.lang.Throwable: Code: 26, e.displayText() = DB::Exception: Cannot parse quoted string: expected opening quote: (at row 1)

Row 1:
Column 0,   name: age,   type: UInt8,         parsed text: "18"
Column 1,   name: money, type: Array(UInt16), parsed text: "[1000,2000]"
Column 2,   name: name,  type: Array(String), parsed text: <EMPTY>ERROR

and if I make some mistakes in purpose, I find that format above is right. I put double quote on each side, and the content parsed are right :

scala> dfb.show
+---+-----------+-----------------+
|age|      money|             name|
+---+-----------+-----------------+
| 18|[1000,2000]|"['liang','fan']"|
+---+-----------+-----------------+

Caused by: java.lang.Throwable: Code: 27, e.displayText() = DB::Exception: Cannot parse input: expected [ before: "[\\\'liang\\\',\\\'fan\\\']"\n: (at row 1)

Row 1:
Column 0,   name: age,   type: UInt8,         parsed text: "18"
Column 1,   name: money, type: Array(UInt16), parsed text: "[1000,2000]"
Column 2,   name: name,  type: Array(String), parsed text: <EMPTY>ERROR

so i do not know why

[doom369]

@Tasselmi hello, have you found the root cause? I'm closing cause it seems it is not driver issue. Feel free to reopen if you think this is a driver issue.