panoramichq / dremio-bigquery-connector Goto Github PK
View Code? Open in Web Editor NEWBigQuery Data Connector for Dremio
License: Apache License 2.0
BigQuery Data Connector for Dremio
License: Apache License 2.0
What would you like to be added:
Described in details what to do with the bigquery json key file
Why is this needed:
I'm trying to configure the connector and I don't understand how to populate the field "Service Account JSON Key". I've tried to copy the content of the file in the field but I've got a generic error message "Failure creating/updating source [sourceName]."
Thank you
Hi, I'm having some trouble with the latest version of the plugin after updating to dremio 15
What happened:
java.lang.NoClassDefFoundError: com/dremio/exec/store/jdbc/JdbcStoragePlugin$Config
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at com.dremio.exec.catalog.conf.ConnectionSchema.findInstantiator(ConnectionSchema.java:87)
at com.dremio.exec.catalog.conf.ConnectionSchema.getSchema(ConnectionSchema.java:80)
at com.dremio.exec.catalog.ConnectionReaderImpl.makeReader(ConnectionReaderImpl.java:75)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.dremio.exec.catalog.ConnectionReader.of(ConnectionReader.java:69)
at com.dremio.dac.daemon.DACDaemonModule.build(DACDaemonModule.java:375)
at com.dremio.dac.daemon.DACDaemon.(DACDaemon.java:186)
at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:313)
at com.dremio.dac.daemon.DACDaemon.newDremioDaemon(DACDaemon.java:321)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:145)
How to reproduce it (as minimally and precisely as possible):
Install plugin with the latest version of Dremio
Details:
What happened:
After filling in values for Name
, Project ID
, Service Account Email Address
, Service Account JSON Key
input fields,
when pressing Save
,
UI warns that Something went wrong
and logs show AbstractMethodError
.
What you expected to happen:
Datasource creation should complete successfully.
How to reproduce it (as minimally and precisely as possible):
Given stock container named dremio
running dremio/dremio-oss:11.0
image,
docker pause dremio
docker cp dremio-bigquery-plugin-0.1.3.jar dremio:/opt/dremio/jars/dremio-bigquery-plugin-0.1.3.jar
docker restart dremio
Anything else we need to know?:
dremio | java.lang.AbstractMethodError: null
dremio | at com.dremio.exec.store.jdbc.conf.JdbcConf.newPlugin(JdbcConf.java:25)
dremio | at com.dremio.exec.store.jdbc.conf.JdbcConf.newPlugin(JdbcConf.java:18)
dremio | at com.dremio.exec.catalog.ManagedStoragePlugin.<init>(ManagedStoragePlugin.java:158)
dremio | at com.dremio.exec.catalog.PluginsManager.newManagedStoragePlugin(PluginsManager.java:268)
dremio | at com.dremio.exec.catalog.PluginsManager.newPlugin(PluginsManager.java:254)
dremio | at com.dremio.exec.catalog.PluginsManager.create(PluginsManager.java:165)
dremio | at com.dremio.exec.catalog.CatalogServiceImpl.createSource(CatalogServiceImpl.java:378)
dremio | at com.dremio.exec.catalog.CatalogServiceImpl.access$400(CatalogServiceImpl.java:101)
dremio | at com.dremio.exec.catalog.CatalogServiceImpl$SourceModifier.createSource(CatalogServiceImpl.java:711)
dremio | at com.dremio.exec.catalog.CatalogImpl.createSource(CatalogImpl.java:964)
dremio | at com.dremio.exec.catalog.SourceAccessChecker.createSource(SourceAccessChecker.java:292)
dremio | at com.dremio.exec.catalog.DelegatingCatalog.createSource(DelegatingCatalog.java:259)
dremio | at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:161)
dremio | at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:152)
dremio | at com.dremio.dac.service.source.SourceService.registerSourceWithRuntime(SourceService.java:148)
dremio | at com.dremio.dac.resource.PutSourceResource.putSource(PutSourceResource.java:80)
dremio | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
dremio | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
dremio | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
dremio | at java.lang.reflect.Method.invoke(Method.java:498)
dremio | at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
dremio | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
dremio | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
dremio | at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
dremio | at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
dremio | at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
dremio | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
dremio | at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
dremio | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
dremio | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
dremio | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
dremio | at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
dremio | at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
dremio | at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
dremio | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
dremio | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
dremio | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
dremio | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
dremio | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
dremio | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
dremio | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
dremio | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
dremio | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
dremio | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
dremio | at com.dremio.dac.server.tracing.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:46)
dremio | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
dremio | at com.dremio.dac.server.GenericResponseHeadersFilter.doFilter(GenericResponseHeadersFilter.java:46)
dremio | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
dremio | at com.dremio.dac.server.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:52)
dremio | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
dremio | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
dremio | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
dremio | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
dremio | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
dremio | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
dremio | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
dremio | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
dremio | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
dremio | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
dremio | at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
dremio | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
dremio | at org.eclipse.jetty.server.Server.handle(Server.java:500)
dremio | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
dremio | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
dremio | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
dremio | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
dremio | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
dremio | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
dremio | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
dremio | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
dremio | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
dremio | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
dremio | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
dremio | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
dremio | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
dremio | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
dremio | at java.lang.Thread.run(Thread.java:748)
What happened:
I have a BigQuery Source, a BigQuery table as a PDS . I am able to setup reflections, refresh reflections are working smooth as expected, when the refresh method is set as full update. But when I set refresh method as Incremental update and select a date column for it, the refresh reflection fails with IllegalArgumentException :
(java.lang.IllegalArgumentException) null com.google.common.base.Preconditions.checkArgument():128 com.dremio.exec.store.jdbc.JdbcRecordReader.checkSchemaConsistency():347 com.dremio.exec.store.jdbc.JdbcRecordReader.setup():219 com.dremio.exec.store.CoercionReader.setup():125 com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():231 com.dremio.sabot.op.scan.ScanOperator.setupReader():204 com.dremio.sabot.op.scan.ScanOperator.setup():190 com.dremio.sabot.driver.SmartOp$SmartProducer.setup():563 com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():79 com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():63 com.dremio.sabot.driver.SmartOp$SmartProducer.accept():533 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.StraightPipe.setup():102 com.dremio.sabot.driver.Pipeline.setup():68 com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():388 com.dremio.sabot.exec.fragment.FragmentExecutor.run():270 com.dremio.sabot.exec.fragment.FragmentExecutor.access$1200():92 com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():679 com.dremio.sabot.task.AsyncTaskWrapper.run():112 com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():226 com.dremio.sabot.task.slicing.SlicingThread.run():156
What you expected to happen:
I am expecting reflection refresh on both the refresh methods , i.e. Full Update and Incremental update. When incremental update is set, I want to select a date column, so newly records with new dates will be the only one to get added in the reflection.
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
I have a 3 node cluster on GCP kubernetes, 2 executors.
Details:
Dremio version (Help > About from the UI): 4.5.0-202006180205030236-1a5d28da
Profile zip:
b149b104-2bd4-490f-b59a-3e8eb463be5a.zip
server.log:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.