GithubHelp home page GithubHelp logo

dremio-bigquery-connector's People

Contributors

mchristopher avatar militarpancho avatar miwelc avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

miwelc mauricioprod maroshmka militarpancho syedqasimahmed praveen-badam sumanentc

dremio-bigquery-connector's Issues

Plugin is not working with new version of Dremio (15.0)

Hi, I'm having some trouble with the latest version of the plugin after updating to dremio 15
What happened:

java.lang.NoClassDefFoundError: com/dremio/exec/store/jdbc/JdbcStoragePlugin$Config
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at com.dremio.exec.catalog.conf.ConnectionSchema.findInstantiator(ConnectionSchema.java:87)
at com.dremio.exec.catalog.conf.ConnectionSchema.getSchema(ConnectionSchema.java:80)
at com.dremio.exec.catalog.ConnectionReaderImpl.makeReader(ConnectionReaderImpl.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.dremio.exec.catalog.ConnectionReader.of(ConnectionReader.java:69)
at com.dremio.dac.daemon.DACDaemonModule.build(DACDaemonModule.java:375)
at com.dremio.dac.daemon.DACDaemon.(DACDaemon.java:186)
at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:313)
at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:321)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:145)

How to reproduce it (as minimally and precisely as possible):
Install plugin with the latest version of Dremio

Details:

  • Dremio version (Help > About from the UI): 15.0.0
  • Profile zip:
  • server.log:

AbstractMethodError

What happened:

After filling in values for Name, Project ID, Service Account Email Address, Service Account JSON Key input fields,
when pressing Save,
UI warns that Something went wrong
and logs show AbstractMethodError.

What you expected to happen:

Datasource creation should complete successfully.

How to reproduce it (as minimally and precisely as possible):

Given stock container named dremio running dremio/dremio-oss:11.0 image,

docker pause dremio
docker cp dremio-bigquery-plugin-0.1.3.jar dremio:/opt/dremio/jars/dremio-bigquery-plugin-0.1.3.jar
docker restart dremio

Anything else we need to know?:

Stack trace from server output 
dremio                  | java.lang.AbstractMethodError: null
dremio                  |       at com.dremio.exec.store.jdbc.conf.JdbcConf.newPlugin(JdbcConf.java:25)
dremio                  |       at com.dremio.exec.store.jdbc.conf.JdbcConf.newPlugin(JdbcConf.java:18)
dremio                  |       at com.dremio.exec.catalog.ManagedStoragePlugin.<init>(ManagedStoragePlugin.java:158)
dremio                  |       at com.dremio.exec.catalog.PluginsManager.newManagedStoragePlugin(PluginsManager.java:268)
dremio                  |       at com.dremio.exec.catalog.PluginsManager.newPlugin(PluginsManager.java:254)
dremio                  |       at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:165)
dremio                  |       at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:378)
dremio                  |       at com.dremio.exec.catalog.CatalogServiceImpl.access$400(CatalogServiceImpl.java:101)
dremio                  |       at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:711)
dremio                  |       at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:964)
dremio                  |       at com.dremio.exec.catalog.SourceAccessChecker.createSource(SourceAccessChecker.java:292)
dremio                  |       at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:259)
dremio                  |       at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:161)
dremio                  |       at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:152)
dremio                  |       at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:148)
dremio                  |       at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80)
dremio                  |       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
dremio                  |       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
dremio                  |       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
dremio                  |       at java.lang.reflect.Method.invoke(Method.java:498)
dremio                  |       at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
dremio                  |       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
dremio                  |       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
dremio                  |       at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
dremio                  |       at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
dremio                  |       at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
dremio                  |       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
dremio                  |       at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
dremio                  |       at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
dremio                  |       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
dremio                  |       at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
dremio                  |       at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
dremio                  |       at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
dremio                  |       at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
dremio                  |       at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
dremio                  |       at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
dremio                  |       at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
dremio                  |       at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
dremio                  |       at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
dremio                  |       at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
dremio                  |       at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
dremio                  |       at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
dremio                  |       at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
dremio                  |       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
dremio                  |       at com.dremio.dac.server.tracing.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:46)
dremio                  |       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
dremio                  |       at com.dremio.dac.server.GenericResponseHeadersFilter.doFilter(GenericResponseHeadersFilter.java:46)
dremio                  |       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
dremio                  |       at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:52)
dremio                  |       at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
dremio                  |       at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
dremio                  |       at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
dremio                  |       at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
dremio                  |       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
dremio                  |       at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
dremio                  |       at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
dremio                  |       at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
dremio                  |       at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
dremio                  |       at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
dremio                  |       at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
dremio                  |       at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
dremio                  |       at org.eclipse.jetty.server.Server.handle(Server.java:500)
dremio                  |       at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
dremio                  |       at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
dremio                  |       at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
dremio                  |       at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
dremio                  |       at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
dremio                  |       at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
dremio                  |       at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
dremio                  |       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
dremio                  |       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
dremio                  |       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
dremio                  |       at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
dremio                  |       at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
dremio                  |       at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
dremio                  |       at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
dremio                  |       at java.lang.Thread.run(Thread.java:748)

Better documentation

What would you like to be added:
Described in details what to do with the bigquery json key file
Why is this needed:
I'm trying to configure the connector and I don't understand how to populate the field "Service Account JSON Key". I've tried to copy the content of the file in the field but I've got a generic error message "Failure creating/updating source [sourceName]."
Thank you

Reflection Refresh: Refresh Method as Incremental Update not working

What happened:
I have a BigQuery Source, a BigQuery table as a PDS . I am able to setup reflections, refresh reflections are working smooth as expected, when the refresh method is set as full update. But when I set refresh method as Incremental update and select a date column for it, the refresh reflection fails with IllegalArgumentException :
(java.lang.IllegalArgumentException) null com.google.common.base.Preconditions.checkArgument():128 com.dremio.exec.store.jdbc.JdbcRecordReader.checkSchemaConsistency():347 com.dremio.exec.store.jdbc.JdbcRecordReader.setup():219 com.dremio.exec.store.CoercionReader.setup():125 com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():231 com.dremio.sabot.op.scan.ScanOperator.setupReader():204 com.dremio.sabot.op.scan.ScanOperator.setup():190 com.dremio.sabot.driver.SmartOp$SmartProducer.setup():563 com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():79 com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():63 com.dremio.sabot.driver.SmartOp$SmartProducer.accept():533 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.Pipeline.setup():68 com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():388 com.dremio.sabot.exec.fragment.FragmentExecutor.run():270 com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():92 com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():679 com.dremio.sabot.task.AsyncTaskWrapper.run():112 com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226 com.dremio.sabot.task.slicing.SlicingThread.run():156

What you expected to happen:
I am expecting reflection refresh on both the refresh methods , i.e. Full Update and Incremental update. When incremental update is set, I want to select a date column, so newly records with new dates will be the only one to get added in the reflection.

How to reproduce it (as minimally and precisely as possible):

  • Create a reflection on a PDS from a BigQuery Table
  • Set refresh method as incremental update, select a column.
  • Click Refresh Now and allow the refresh to happen. This refresh will fail, you can check on the jobs page
    image

Anything else we need to know?:
I have a 3 node cluster on GCP kubernetes, 2 executors.

Details:

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.