vabezruchko / spark-clickhouse-connector Goto Github PK
View Code? Open in Web Editor NEWSpark Clickhouse Connector
License: Apache License 2.0
Spark Clickhouse Connector
License: Apache License 2.0
In this method: io.clickhouse.spark.connector.SparkClickhouseFunctions#clickhouseTable(query: String, cluster: String)
parameter cluster
is not optional.
Is there a way (or workaround) to read from ClickHouse single server.
Sparkcluster must be needed?
How about the batch data analysis performance of hundreds of millions of data, do you have test result data?
Where to set the username and password param when connect to my clickhouse ?
Easy to use,plan support spark3 extension integration clickhouse?
When use driver"com.github.housepower.jdbc.ClickHouseDriver" and URL over 9000,the problem below occurs.
15996 [Executor task launch worker for task 0] ERROR org.apache.spark.executor.Executor - Exception in task 0.0 in stage 0.0 (TID 0)
java.sql.SQLFeatureNotSupportedException
at com.github.housepower.jdbc.wrapper.SQLResultSetMetaData.getPrecision(SQLResultSetMetaData.java:65)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(JdbcUtils.scala:296)
at org.apache.spark.sql.ClickhouseRowFactory$.apply(ClickhouseRow.scala:35)
at io.clickhouse.spark.connector.ClickhouseRDD.compute(ClickhouseRDD.scala:125)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
Is there any function to save DataFrame/RDD to ClickHouseDB ?
Just after installing sbt I am getting this error. I understand this is because it wants to publish it to Nexus Repository Manager, but the question is: I don't have any, and I don't need it for my experimental machine.
Could you please do some option to remove it?
[info] Non-compiled module 'compiler-bridge_2.12' for Scala 2.12.7. Compiling...
[info] Compilation completed in 4.391s.
[info] Done compiling.
[error] java.lang.ExceptionInInitializerError
[error] at java.lang.Class.forName0(Native Method)
[error] at java.lang.Class.forName(Class.java:348)
[error] at sbt.compiler.Eval$.getModule(Eval.scala:584)
[error] at sbt.compiler.EvalDefinitions.values(Eval.scala:58)
[error] at sbt.internal.DefinedSbtValues.$anonfun$values$1(LoadedSbtFile.scala:50)
[error] at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
[error] at scala.collection.immutable.List.foreach(List.scala:388)
[error] at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
[error] at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
[error] at scala.collection.immutable.List.flatMap(List.scala:351)
[error] at sbt.internal.DefinedSbtValues.values(LoadedSbtFile.scala:50)
[error] at sbt.internal.EvaluateConfigurations$.$anonfun$evaluateSbtFile$4(EvaluateConfigurations.scala:166)
[error] at sbt.internal.Load$.loadSettingsFile$1(Load.scala:1137)
[error] at sbt.internal.Load$.$anonfun$discoverProjects$2(Load.scala:1144)
[error] at scala.collection.MapLike.getOrElse(MapLike.scala:127)
[error] at scala.collection.MapLike.getOrElse$(MapLike.scala:125)
[error] at scala.collection.AbstractMap.getOrElse(Map.scala:59)
[error] at sbt.internal.Load$.memoLoadSettingsFile$1(Load.scala:1143)
[error] at sbt.internal.Load$.$anonfun$discoverProjects$4(Load.scala:1151)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
[error] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
[error] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
[error] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:233)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
[error] at scala.collection.AbstractTraversable.map(Traversable.scala:104)
[error] at sbt.internal.Load$.loadFiles$1(Load.scala:1151)
[error] at sbt.internal.Load$.discoverProjects(Load.scala:1165)
[error] at sbt.internal.Load$.discover$1(Load.scala:862)
[error] at sbt.internal.Load$.loadTransitive(Load.scala:937)
[error] at sbt.internal.Load$.loadProjects$1(Load.scala:726)
[error] at sbt.internal.Load$.$anonfun$loadUnit$11(Load.scala:729)
[error] at sbt.internal.Load$.timed(Load.scala:1395)
[error] at sbt.internal.Load$.$anonfun$loadUnit$1(Load.scala:729)
[error] at sbt.internal.Load$.timed(Load.scala:1395)
[error] at sbt.internal.Load$.loadUnit(Load.scala:688)
[error] at sbt.internal.Load$.$anonfun$builtinLoader$4(Load.scala:484)
[error] at sbt.internal.BuildLoader$.$anonfun$componentLoader$5(BuildLoader.scala:176)
[error] at sbt.internal.BuildLoader.apply(BuildLoader.scala:241)
[error] at sbt.internal.Load$.loadURI$1(Load.scala:546)
[error] at sbt.internal.Load$.loadAll(Load.scala:562)
[error] at sbt.internal.Load$.loadURI(Load.scala:492)
[error] at sbt.internal.Load$.load(Load.scala:471)
[error] at sbt.internal.Load$.$anonfun$apply$1(Load.scala:251)
[error] at sbt.internal.Load$.timed(Load.scala:1395)
[error] at sbt.internal.Load$.apply(Load.scala:251)
[error] at sbt.internal.Load$.defaultLoad(Load.scala:69)
[error] at sbt.BuiltinCommands$.liftedTree1$1(Main.scala:829)
[error] at sbt.BuiltinCommands$.doLoadProject(Main.scala:829)
[error] at sbt.BuiltinCommands$.$anonfun$loadProjectImpl$2(Main.scala:800)
[error] at sbt.Command$.$anonfun$applyEffect$4(Command.scala:142)
[error] at sbt.Command$.$anonfun$applyEffect$2(Command.scala:137)
[error] at sbt.Command$.process(Command.scala:181)
[error] at sbt.MainLoop$.processCommand(MainLoop.scala:151)
[error] at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:139)
[error] at sbt.State$$anon$1.runCmd$1(State.scala:246)
[error] at sbt.State$$anon$1.process(State.scala:250)
[error] at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:139)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.MainLoop$.next(MainLoop.scala:139)
[error] at sbt.MainLoop$.run(MainLoop.scala:132)
[error] at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:110)
[error] at sbt.io.Using.apply(Using.scala:22)
[error] at sbt.MainLoop$.runWithNewLog(MainLoop.scala:104)
[error] at sbt.MainLoop$.runAndClearLast(MainLoop.scala:59)
[error] at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:44)
[error] at sbt.MainLoop$.runLogged(MainLoop.scala:35)
[error] at sbt.StandardMain$.runManaged(Main.scala:138)
[error] at sbt.xMain.run(Main.scala:89)
[error] at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:111)
[error] at xsbt.boot.Launch$.withContextLoader(Launch.scala:131)
[error] at xsbt.boot.Launch$.run(Launch.scala:111)
[error] at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:37)
[error] at xsbt.boot.Launch$.launch(Launch.scala:120)
[error] at xsbt.boot.Launch$.apply(Launch.scala:20)
[error] at xsbt.boot.Boot$.runImpl(Boot.scala:56)
[error] at xsbt.boot.Boot$.main(Boot.scala:18)
[error] at xsbt.boot.Boot.main(Boot.scala)
[error] Caused by: java.io.FileNotFoundException: C:\Users\Ihor_Bobak\.sbt\nexus_url (The system cannot find the file specified)
[error] at java.io.FileInputStream.open0(Native Method)
[error] at java.io.FileInputStream.open(FileInputStream.java:195)
[error] at java.io.FileInputStream.<init>(FileInputStream.java:138)
[error] at sbt.io.Using$.$anonfun$fileInputStream$1(Using.scala:88)
[error] at sbt.io.Using$$anon$2.openImpl(Using.scala:76)
[error] at sbt.io.OpenFile.open(Using.scala:43)
[error] at sbt.io.OpenFile.open$(Using.scala:39)
[error] at sbt.io.Using$$anon$2.open(Using.scala:75)
[error] at sbt.io.Using$$anon$2.open(Using.scala:75)
[error] at sbt.io.Using.apply(Using.scala:21)
[error] at sbt.io.IO$.transfer(IO.scala:436)
[error] at sbt.io.IO$.read(IO.scala:892)
[error] at $a7ade90dd9048d56449d$.<init>(build.sbt:8)
[error] at $a7ade90dd9048d56449d$.<clinit>(build.sbt)
[error] at java.lang.Class.forName0(Native Method)
[error] at java.lang.Class.forName(Class.java:348)
[error] at sbt.compiler.Eval$.getModule(Eval.scala:584)
[error] at sbt.compiler.EvalDefinitions.values(Eval.scala:58)
[error] at sbt.internal.DefinedSbtValues.$anonfun$values$1(LoadedSbtFile.scala:50)
[error] at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:240)
[error] at scala.collection.immutable.List.foreach(List.scala:388)
[error] at scala.collection.TraversableLike.flatMap(TraversableLike.scala:240)
[error] at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:237)
[error] at scala.collection.immutable.List.flatMap(List.scala:351)
[error] at sbt.internal.DefinedSbtValues.values(LoadedSbtFile.scala:50)
[error] at sbt.internal.EvaluateConfigurations$.$anonfun$evaluateSbtFile$4(EvaluateConfigurations.scala:166)
[error] at sbt.internal.Load$.loadSettingsFile$1(Load.scala:1137)
[error] at sbt.internal.Load$.$anonfun$discoverProjects$2(Load.scala:1144)
[error] at scala.collection.MapLike.getOrElse(MapLike.scala:127)
[error] at scala.collection.MapLike.getOrElse$(MapLike.scala:125)
[error] at scala.collection.AbstractMap.getOrElse(Map.scala:59)
[error] at sbt.internal.Load$.memoLoadSettingsFile$1(Load.scala:1143)
[error] at sbt.internal.Load$.$anonfun$discoverProjects$4(Load.scala:1151)
[error] at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
[error] at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
[error] at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
[error] at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
[error] at scala.collection.TraversableLike.map(TraversableLike.scala:233)
[error] at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
[error] at scala.collection.AbstractTraversable.map(Traversable.scala:104)
[error] at sbt.internal.Load$.loadFiles$1(Load.scala:1151)
[error] at sbt.internal.Load$.discoverProjects(Load.scala:1165)
[error] at sbt.internal.Load$.discover$1(Load.scala:862)
[error] at sbt.internal.Load$.loadTransitive(Load.scala:937)
[error] at sbt.internal.Load$.loadProjects$1(Load.scala:726)
[error] at sbt.internal.Load$.$anonfun$loadUnit$11(Load.scala:729)
[error] at sbt.internal.Load$.timed(Load.scala:1395)
[error] at sbt.internal.Load$.$anonfun$loadUnit$1(Load.scala:729)
[error] at sbt.internal.Load$.timed(Load.scala:1395)
[error] at sbt.internal.Load$.loadUnit(Load.scala:688)
[error] at sbt.internal.Load$.$anonfun$builtinLoader$4(Load.scala:484)
[error] at sbt.internal.BuildLoader$.$anonfun$componentLoader$5(BuildLoader.scala:176)
[error] at sbt.internal.BuildLoader.apply(BuildLoader.scala:241)
[error] at sbt.internal.Load$.loadURI$1(Load.scala:546)
[error] at sbt.internal.Load$.loadAll(Load.scala:562)
[error] at sbt.internal.Load$.loadURI(Load.scala:492)
[error] at sbt.internal.Load$.load(Load.scala:471)
[error] at sbt.internal.Load$.$anonfun$apply$1(Load.scala:251)
[error] at sbt.internal.Load$.timed(Load.scala:1395)
[error] at sbt.internal.Load$.apply(Load.scala:251)
[error] at sbt.internal.Load$.defaultLoad(Load.scala:69)
[error] at sbt.BuiltinCommands$.liftedTree1$1(Main.scala:829)
[error] at sbt.BuiltinCommands$.doLoadProject(Main.scala:829)
[error] at sbt.BuiltinCommands$.$anonfun$loadProjectImpl$2(Main.scala:800)
[error] at sbt.Command$.$anonfun$applyEffect$4(Command.scala:142)
[error] at sbt.Command$.$anonfun$applyEffect$2(Command.scala:137)
[error] at sbt.Command$.process(Command.scala:181)
[error] at sbt.MainLoop$.processCommand(MainLoop.scala:151)
[error] at sbt.MainLoop$.$anonfun$next$2(MainLoop.scala:139)
[error] at sbt.State$$anon$1.runCmd$1(State.scala:246)
[error] at sbt.State$$anon$1.process(State.scala:250)
[error] at sbt.MainLoop$.$anonfun$next$1(MainLoop.scala:139)
[error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:16)
[error] at sbt.MainLoop$.next(MainLoop.scala:139)
[error] at sbt.MainLoop$.run(MainLoop.scala:132)
[error] at sbt.MainLoop$.$anonfun$runWithNewLog$1(MainLoop.scala:110)
[error] at sbt.io.Using.apply(Using.scala:22)
[error] at sbt.MainLoop$.runWithNewLog(MainLoop.scala:104)
[error] at sbt.MainLoop$.runAndClearLast(MainLoop.scala:59)
[error] at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:44)
[error] at sbt.MainLoop$.runLogged(MainLoop.scala:35)
[error] at sbt.StandardMain$.runManaged(Main.scala:138)
[error] at sbt.xMain.run(Main.scala:89)
[error] at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:111)
[error] at xsbt.boot.Launch$.withContextLoader(Launch.scala:131)
[error] at xsbt.boot.Launch$.run(Launch.scala:111)
[error] at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:37)
[error] at xsbt.boot.Launch$.launch(Launch.scala:120)
[error] at xsbt.boot.Launch$.apply(Launch.scala:20)
[error] at xsbt.boot.Boot$.runImpl(Boot.scala:56)
[error] at xsbt.boot.Boot$.main(Boot.scala:18)
[error] at xsbt.boot.Boot.main(Boot.scala)
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? [error] java.lang.ExceptionInInitializerError
[error] Use 'last' for the full log.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.