can not read from BQ table about spark-bigquery-connector

[pmkc]

com.google.api.gax.* should be shaded/relocated in our connector, but isn't shaded in your stack trace. Are you using gs://spark-lib/bigquery/spark-bigquery-latest.jar or building it yourself?

If you build it yourself you should use sbt assembly.
If you are compiling against it I would mark it provided and use our shaded jar.

[pmkc]

This is the same as #36. I think I will update the compilation instructions to compile against the seaded profile.

[JD-V]

@pmkc I am building it myself using sbt assembly, even though I am seeing this error.

[pmkc]

Can you give me your approximate build and run commands.
If you are compiling against the connector could you show me your build.sbt?

[JD-V]

my build command is sbt assembly

and this is how my build.sbt looks like,

name := "sparklib"

version := "0.1"

scalaVersion := "2.11.12"


libraryDependencies += "org.apache.spark" %% "spark-core" % "2.4.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.0" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.4.0" % "provided"
libraryDependencies += "com.typesafe" % "config" % "1.3.4" % "provided"

libraryDependencies += "log4j" % "log4j" % "1.2.15" excludeAll( ExclusionRule(organization = "com.sun.jdmk"), ExclusionRule(organization = "com.sun.jmx"), ExclusionRule(organization = "javax.jms") )

libraryDependencies += "com.google.cloud.bigdataoss" % "gcs-connector" % "hadoop2-1.9.17"  % "provided"
libraryDependencies += "com.google.cloud.bigdataoss" % "bigquery-connector" % "hadoop2-0.13.5" % "provided"
libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery" % "0.7.0-beta"


resolvers += Opts.resolver.sonatypeReleases

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.common.**" -> "repackaged.com.google.common.@1").inAll,
  ShadeRule.rename("com.google.protobuf.**" -> "repackaged.com.google.protobuf.@1").inAll
)

mainClass in assembly := some("LoadData")
assemblyJarName := "assembly_sparklib_2.11-0.1.jar"

[achelimed]

Could you try this?

 ShadeRule.rename("com.google.guava.**"        -> "repackaged.com.google.guava.@1").inAll,
  ShadeRule.rename("com.google.common.guava.**" -> "repackaged.com.google.common.guava.@1").inAll,
  ShadeRule.rename("com.google.protobuf.**"     -> "repackaged.com.google.protobuf.@1").inAll

[pmkc]

I'm not actually sure why that is giving you a no such method for grpc, you could try sbt dependencyTree to find a conflict, but I would just side step that.

I would just mark "spark-bigquery" provided and use gs://spark-lib/bigquery/spark-bigquery-latest.jar OR compile against the shaded qualifier.

[JD-V]

@achelimed I am seeing same error even after applying shadeRules you mentioned above.

[JD-V]

This issue seems to be resolved in 0.8.0 release. Thank you @achelimed and @pmkc for your help.