tabular-io / iceberg-rest-image Goto Github PK
View Code? Open in Web Editor NEWSimple project to expose a catalog over REST using a Java catalog backend
License: Apache License 2.0
iceberg-rest-image's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
abmo-x dramaticlly bicepforbigdataengineer georgehu0815 bangpound as10128 fractal-dynamics cccs-eric ralnoc ln-abourayaa liqinrae saoxia kevinjqliu alexrogalskiy vspyrop wu-rds alexjo2144 slachiewicz jqin61 nsftx trmlabs westse skhatri vakarisbk amitgilad3 gavrmike sullis mindok7520 ericsun2 krishhkk kiersten-stokes dashbook esmivn pcorrea-trm ivenkateswarlu yassaniceberg-rest-image's Issues
Health check for Kubernetes
Is there any possibility of having a health check for use in Kubernetes? We've seen that when using iceberg-rest with PostgreSQL if the connection to the database server drops, the iceberg-rest server needs rebooting. This could be easily handled automatically from Kubernetes by having a health check but I don't know how much effort it would be to add in health checks for each catalog (where necessary)
No NessieCatalog?
Setting CATALOG_IO__IMPL=org.apache.iceberg.nessie.NessieCatalog throws an exception immediately at startup:
2023-01-18T15:54:22.583 INFO [org.apache.iceberg.CatalogUtil] - Loading custom FileIO implementation: org.apache.iceberg.nessie.NessieCatalog
Exception in thread "main" java.lang.IllegalArgumentException: Cannot initialize FileIO, missing no-arg constructor: org.apache.iceberg.nessie.NessieCatalog
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:311)
at org.apache.iceberg.jdbc.JdbcCatalog.initialize(JdbcCatalog.java:101)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:238)
at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:283)
at org.apache.iceberg.rest.RESTCatalogServer.backendCatalog(RESTCatalogServer.java:83)
at org.apache.iceberg.rest.RESTCatalogServer.main(RESTCatalogServer.java:87)
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.io.FileIO
Missing org.apache.iceberg.nessie.NessieCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.nessie.NessieCatalog]
at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250)
at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32)
at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220)
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:308)
... 5 more
Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.nessie.NessieCatalog
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149)
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:307)
... 5 more
No suitable driver found for jdbc:mysql
Hi. I'm attempting to use a MySQL database instead of SQLite, which is the default for the catalog database. However, it appears that there is no MySQL driver class available, as shown below in the Apache Iceberg documentation:
Class.forName("com.mysql.cj.jdbc.Driver"); // ensure JDBC driver is at runtime classpath
Map<String, String> properties = new HashMap<>();
properties.put(CatalogProperties.CATALOG_IMPL, JdbcCatalog.class.getName());
properties.put(CatalogProperties.URI, "jdbc:mysql://localhost:3306/test");
properties.put(JdbcCatalog.PROPERTY_PREFIX + "user", "admin");
properties.put(JdbcCatalog.PROPERTY_PREFIX + "password", "pass");
properties.put(CatalogProperties.WAREHOUSE_LOCATION, "s3://warehouse/path");
Configuration hadoopConf = new Configuration(); // configs if you use HadoopFileIO
JdbcCatalog catalog = CatalogUtil.buildIcebergCatalog("test_jdbc_catalog", properties, hadoopConf);When I changed the environment like this: "CATALOG_URI: "jdbc:mysql://10.10.50.182/lakehouse", the logs output is as follows:
Exception in thread "main" org.apache.iceberg.jdbc.UncheckedSQLException: Failed to connect:
jdbc:mysql://10.10.50.182/lakehouse
at org.apache.iceberg.jdbc.JdbcClientPool.newClient(JdbcClientPool.java:57)
at org.apache.iceberg.jdbc.JdbcClientPool.newClient(JdbcClientPool.java:30)
at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:125)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.jdbc.JdbcCatalog.initializeCatalogTables(JdbcCatalog.java:155)
at org.apache.iceberg.jdbc.JdbcCatalog.initialize(JdbcCatalog.java:141)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:255)
at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:309)
at org.apache.iceberg.rest.RESTCatalogServer.backendCatalog(RESTCatalogServer.java:86)
at org.apache.iceberg.rest.RESTCatalogServer.main(RESTCatalogServer.java:90)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:mysql://10.10.50.182/lakehouse
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:706)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
at org.apache.iceberg.jdbc.JdbcClientPool.newClient(JdbcClientPool.java:55)
... 10 more
Do you have any plans to address this issue? Thank you.
No s3 support for writing table metadata
From trino, I can create the schema/namespace. On CREATE TABLE, writing the underlying files to s3 (actually, minio) works, but on committing the metadata, thows an exception:
2023-01-18T15:47:10.326 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] - Error processing REST request
org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=500, type=RuntimeIOException, message=Failed to get file system for path: s3://testdata/tinynation/metadata/00000-00b584bb-e702-4484-9968-92c0948f08ca.metadata.json)
org.apache.iceberg.exceptions.RuntimeIOException: Failed to get file system for path: s3://testdata/tinynation/metadata/00000-00b584bb-e702-4484-9968-92c0948f08ca.metadata.json
at org.apache.iceberg.hadoop.Util.getFs(Util.java:54)
at org.apache.iceberg.hadoop.HadoopOutputFile.fromPath(HadoopOutputFile.java:53)
at org.apache.iceberg.hadoop.HadoopFileIO.newOutputFile(HadoopFileIO.java:83)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:159)
at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:105)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.rest.CatalogHandlers.create(CatalogHandlers.java:315)
at org.apache.iceberg.rest.CatalogHandlers.updateTable(CatalogHandlers.java:263)
at org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:343)
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:384)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.iceberg.hadoop.Util.getFs(Util.java:52)
... 42 more
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:401)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)
I think this is a config issue, but I haven't been able to suss out what env vars I need to declare.
How to run REST Catalog Server with OAuth2 Authentication?
I want to add iceberg REST catalog with the value of OAUTH2 for the option of iceberg.rest-catalog.security to my trino cluster, for which REST catalog server needs to be run with support of OAuth2 authentication.
I have added some codes at the end of RESTCatalogServer like this.
RESTCatalogAdapter adapter = new RESTCatalogAdapter(backendCatalog());
RESTCatalogServlet servlet = new RESTCatalogServlet(adapter);
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
context.setContextPath("/");
ServletHolder servletHolder = new ServletHolder(servlet);
servletHolder.setInitParameter("javax.ws.rs.Application", "ServiceListPublic");
context.addServlet(servletHolder, "/*");
context.setVirtualHosts(null);
context.setGzipHandler(new GzipHandler());
Server httpServer =
new Server(PropertyUtil.propertyAsInt(System.getenv(), "REST_PORT", 8181));
httpServer.setHandler(context);
httpServer.start();
// ADDED CODES!!!!!!!
SessionCatalog.SessionContext sessionContext =
new SessionCatalog.SessionContext(
UUID.randomUUID().toString(),
"user",
ImmutableMap.of("credential", "user:secret"),
ImmutableMap.of());
RESTCatalog catalog = new RESTCatalog(sessionContext, (config) -> adapter);
catalog.initialize(
"prod", ImmutableMap.of(CatalogProperties.URI, "ignored", "token", "bearer-token"));
But it seems, this is not OAuth2 enabled server.
Anybody knows how to add OAuth2 authentication to RESTCatalogServer?
Support for hivecatalog
Is there a way to connect to a Hive Metastore Catalog for this rest interface?
if not, will it be added it in the future?
Thanks!
official helm chart or operator to deploy HA & scalable REST Catalog on kubernetes
hi, is there a well-tested way to deploy REST Catalog on kubernetes? great thanks.
Image doesn't use in-memory database per default
The documentation of this image on DockerHub mentions that it uses an 'in memory backend'. However, I tested it locally and noticed that the catalog was in fact persisted (i.e. it still existed after rebooting the container). I assume this is due to an error in the source code.
I think the connection string jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory in
catalogProperties.putIfAbsent(
CatalogProperties.URI, "jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory");needs to be replaced. Currently, this creates an iceberg_rest_mode=memory file in the tmp directory of the container that is persisted and is just a regular sqlite DB file.
How to use HadoopFileIO as CATALOG_IMPL
Hello,
I was using the example project docker-spark-iceberg in a Docker environment and I've to replace the org.apache.iceberg.aws.s3.S3FileIO with org.apache.iceberg.hadoop.HadoopFileIO for legacy problems with s3a schema.
I've the following error:
22/11/17 13:36:37 ERROR Utils: Aborting task
org.apache.iceberg.exceptions.ServiceFailureException: Server error: RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.iceberg.rest.ErrorHandlers.lambda$defaultErrorHandler$3(ErrorHandlers.java:125)
at java.base/java.util.function.Consumer.lambda$andThen$0(Consumer.java:65)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:129)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:193)
at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:228)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:68)
at org.apache.iceberg.rest.RESTTableOperations.commit(RESTTableOperations.java:130)
at org.apache.iceberg.BaseTransaction.commitCreateTransaction(BaseTransaction.java:277)
at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:256)
at org.apache.iceberg.spark.source.StagedSparkTable.commitStagedChanges(StagedSparkTable.java:35)
at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.$anonfun$writeToTable$1(WriteToDataSourceV2Exec.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable(WriteToDataSourceV2Exec.scala:491)
at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable$(WriteToDataSourceV2Exec.scala:486)
at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.writeToTable(WriteToDataSourceV2Exec.scala:108)
at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.run(WriteToDataSourceV2Exec.scala:131)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:636)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:566)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
How can I add the hadoop-aws(https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws) dependency to the RESTCatalog?
Thanks!
License missing
Hi,
some of your files do contain a license header but not all.
This makes it problematic to use.
Would you be willing to add a LICENSE to the repository?
Consider limiting iceberg client pool size for default sqlite backend
Hi team,
thanks for this very useful project. I've been having good success testing various things with iceberg using this catalog.
Recently, I've been getting 500s under somewhat heavy load:
2023-08-10T14:44:52.707 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] - Error processing REST request
org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=500, type=UncheckedSQLException, message=Unknown failure)
I was able to find the following in the stack trace:
Caused by: org.sqlite.SQLiteException: [SQLITE_BUSY] The database file is locked (database is locked)
at org.sqlite.core.DB.newSQLException(DB.java:1179)
at org.sqlite.core.DB.newSQLException(DB.java:1190)
at org.sqlite.core.DB.execute(DB.java:985)
at org.sqlite.core.DB.executeUpdate(DB.java:1054)
at org.sqlite.jdbc3.JDBC3PreparedStatement.lambda$executeLargeUpdate$2(JDBC3PreparedStatement.java:118) iceberg-rest-s3-aws | at org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:454)
at org.sqlite.jdbc3.JDBC3PreparedStatement.executeLargeUpdate(JDBC3PreparedStatement.java:117)
at org.sqlite.jdbc3.JDBC3PreparedStatement.executeUpdate(JDBC3PreparedStatement.java:99)
at org.apache.iceberg.jdbc.JdbcTableOperations.lambda$updateTable$0(JdbcTableOperations.java:164)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.jdbc.JdbcTableOperations.updateTable(JdbcTableOperations.java:153)
at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:115)
... 43 more
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:401)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
My guess was that there is some sort of connection pooling of JDBC connections going on and sqlite doesn't play nice with that. I checked the source of JdbcClientPool and saw that the default pool size is 2, this increased my suspicions.
I set it to 1 using CATALOG_CLIENTS=1 and haven't had these errors since.
I propose to set this to 1 if the default sqlite JDBC connection string is used. I'd be happy to draft a PR if someone else can confirm my suspicions.
Cheers,
Johannes
CVE security findings that need to be resolved
Security findings found via Trivy when using that need to be resolved. Unsure if this should be reported here or upstream.
In summary there are 5 medium and 4 high CVE vulnerabilities that can be resolved with updated dependencies within iceberg-rest-image-all.jar
Note that this is with the following contents in build.gradle
ext {
icebergVersion = '1.5.2'
hadoopVersion = '3.4.0'
}
Total: 39 (UNKNOWN: 0, LOW: 33, MEDIUM: 6, HIGH: 0, CRITICAL: 0)
┌──────────────────┬────────────────┬──────────┬──────────┬──────────────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├──────────────────┼────────────────┼──────────┼──────────┼──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ coreutils │ CVE-2016-2781 │ LOW │ affected │ 8.32-4.1ubuntu1.2 │ │ coreutils: Non-privileged session can escape to the parent │
│ │ │ │ │ │ │ session in chroot │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-2781 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ gcc-12-base │ CVE-2022-27943 │ │ │ 12.3.0-1ubuntu1~22.04 │ │ binutils: libiberty/rust-demangle.c in GNU GCC 11.2 allows │
│ │ │ │ │ │ │ stack exhaustion in demangle_const │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-27943 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ gpgv │ CVE-2022-3219 │ │ │ 2.2.27-3ubuntu2.1 │ │ gnupg: denial of service issue (resource consumption) using │
│ │ │ │ │ │ │ compressed packets │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-3219 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libc-bin │ CVE-2016-20013 │ │ │ 2.35-0ubuntu3.7 │ │ sha256crypt and sha512crypt through 0.6 allow attackers to │
│ │ │ │ │ │ │ cause a denial of... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-20013 │
├──────────────────┤ │ │ │ ├───────────────┤ │
│ libc6 │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libgcc-s1 │ CVE-2022-27943 │ │ │ 12.3.0-1ubuntu1~22.04 │ │ binutils: libiberty/rust-demangle.c in GNU GCC 11.2 allows │
│ │ │ │ │ │ │ stack exhaustion in demangle_const │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-27943 │
├──────────────────┼────────────────┼──────────┤ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libgcrypt20 │ CVE-2024-2236 │ MEDIUM │ │ 1.9.4-3ubuntu3 │ │ libgcrypt: vulnerable to Marvin Attack │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-2236 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libgssapi-krb5-2 │ CVE-2024-26462 │ │ │ 1.19.2-2ubuntu0.3 │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libk5crypto3 │ CVE-2024-26462 │ MEDIUM │ │ │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libkrb5-3 │ CVE-2024-26462 │ MEDIUM │ │ │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libkrb5support0 │ CVE-2024-26462 │ MEDIUM │ │ │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ liblzma5 │ CVE-2020-22916 │ MEDIUM │ │ 5.2.5-2ubuntu1 │ │ Denial of service via decompression of crafted file │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2020-22916 │
├──────────────────┼────────────────┼──────────┤ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libncurses6 │ CVE-2023-45918 │ LOW │ │ 6.3-2ubuntu0.1 │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libncursesw6 │ CVE-2023-45918 │ │ │ │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libpcre3 │ CVE-2017-11164 │ │ │ 2:8.39-13ubuntu0.22.04.1 │ │ pcre: OP_KETRMAX feature in the match function in │
│ │ │ │ │ │ │ pcre_exec.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2017-11164 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libpng16-16 │ CVE-2022-3857 │ │ │ 1.6.37-3build5 │ │ libpng: Null pointer dereference leads to segmentation fault │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-3857 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libstdc++6 │ CVE-2022-27943 │ │ │ 12.3.0-1ubuntu1~22.04 │ │ binutils: libiberty/rust-demangle.c in GNU GCC 11.2 allows │
│ │ │ │ │ │ │ stack exhaustion in demangle_const │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-27943 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libsystemd0 │ CVE-2023-7008 │ │ │ 249.11-0ubuntu3.12 │ │ systemd-resolved: Unsigned name response in signed zone is │
│ │ │ │ │ │ │ not refused when DNSSEC=yes... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-7008 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libtinfo6 │ CVE-2023-45918 │ │ │ 6.3-2ubuntu0.1 │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libudev1 │ CVE-2023-7008 │ │ │ 249.11-0ubuntu3.12 │ │ systemd-resolved: Unsigned name response in signed zone is │
│ │ │ │ │ │ │ not refused when DNSSEC=yes... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-7008 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libzstd1 │ CVE-2022-4899 │ │ │ 1.4.8+dfsg-3build1 │ │ zstd: mysql: buffer overrun in util.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-4899 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ locales │ CVE-2016-20013 │ │ │ 2.35-0ubuntu3.7 │ │ sha256crypt and sha512crypt through 0.6 allow attackers to │
│ │ │ │ │ │ │ cause a denial of... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-20013 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ login │ CVE-2023-29383 │ │ │ 1:4.8.1-2ubuntu2.2 │ │ shadow: Improper input validation in shadow-utils package │
│ │ │ │ │ │ │ utility chfn │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-29383 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ ncurses-base │ CVE-2023-45918 │ │ │ 6.3-2ubuntu0.1 │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ ncurses-bin │ CVE-2023-45918 │ │ │ │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ passwd │ CVE-2023-29383 │ │ │ 1:4.8.1-2ubuntu2.2 │ │ shadow: Improper input validation in shadow-utils package │
│ │ │ │ │ │ │ utility chfn │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-29383 │
└──────────────────┴────────────────┴──────────┴──────────┴──────────────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘
2024-05-29T11:12:12-04:00 INFO Table result includes only package filenames. Use '--format json' option to get the full path to the package file.
Java (jar)
Total: 9 (UNKNOWN: 0, LOW: 0, MEDIUM: 5, HIGH: 4, CRITICAL: 0)
┌─────────────────────────────────────────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬────────────────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ ch.qos.logback:logback-classic (iceberg-rest-image-all.jar) │ CVE-2023-6378 │ HIGH │ fixed │ 1.2.10 │ 1.3.12, 1.4.12, 1.2.13 │ logback: serialization vulnerability in logback receiver │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-6378 │
├─────────────────────────────────────────────────────────────┤ │ │ │ │ │ │
│ ch.qos.logback:logback-core (iceberg-rest-image-all.jar) │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┤ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ com.nimbusds:nimbus-jose-jwt (iceberg-rest-image-all.jar) │ CVE-2023-52428 │ MEDIUM │ │ 9.30.2 │ 9.37.2 │ Denial of Service in Connect2id Nimbus JOSE+JWT │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-52428 │
├─────────────────────────────────────────────────────────────┼────────────────┤ │ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ io.netty:netty-codec-http (iceberg-rest-image-all.jar) │ CVE-2024-29025 │ │ │ 4.1.100.Final │ 4.1.108.Final │ netty-codec-http: Allocation of Resources Without Limits or │
│ │ │ │ │ │ │ Throttling │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-29025 │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┤ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ org.apache.commons:commons-compress │ CVE-2024-25710 │ HIGH │ │ 1.24.0 │ 1.26.0 │ commons-compress: Denial of service caused by an infinite │
│ (iceberg-rest-image-all.jar) │ │ │ │ │ │ loop for a corrupted... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-25710 │
│ ├────────────────┤ │ │ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26308 │ │ │ │ │ commons-compress: OutOfMemoryError unpacking broken Pack200 │
│ │ │ │ │ │ │ file │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26308 │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┤ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ org.apache.commons:commons-configuration2 │ CVE-2024-29131 │ MEDIUM │ │ 2.8.0 │ 2.10.1 │ commons-configuration: StackOverflowError adding property in │
│ (iceberg-rest-image-all.jar) │ │ │ │ │ │ AbstractListDelimiterHandler.flattenIterator() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-29131 │
│ ├────────────────┤ │ │ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-29133 │ │ │ │ │ commons-configuration: StackOverflowError calling │
│ │ │ │ │ │ │ ListDelimiterHandler.flatten(Object, int) with a cyclical │
│ │ │ │ │ │ │ object tree │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-29133 │
├─────────────────────────────────────────────────────────────┼────────────────┤ │ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ org.apache.zookeeper:zookeeper (iceberg-rest-image-all.jar) │ CVE-2024-23944 │ │ │ 3.8.3 │ 3.8.4, 3.9.2 │ Information disclosure in persistent watchers handling in │
│ │ │ │ │ │ │ Apache ZooKe ... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-23944 │
└─────────────────────────────────────────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴────────────────────────┴──────────────────────────────────────────────────────────────┘
GCS support
Hi guys,
Are you going to add GCS support?
Any ETA?
Can we manage multiple catalogs ?
When using Jdbc Catalog, I can have many catalogs as I want. Each catalog contains its own namespaces and tables.
But with REST catalog, I wonder if I can do the same thing like JDBC Catalog. I can't find any API to create catalog and show all catalogs.
Can someone give me an ideas?
Thank!
Unable to run rest-catalog with postgres instead of sqlite
Im running this docker compose file:
---
version: "3"
services:
iceberg-postgres-catalog:
build:
context: .
image: postgres:latest
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_USER: icebergcatalog
POSTGRES_PASSWORD: icebergcatalog
POSTGRES_DB: iceberg_catalog
iceberg-rest-catalog:
image: tabulario/iceberg-rest:latest
ports:
- "8181:8181"
environment:
CATALOG_URI: jdbc:postgres:/iceberg-postgres-catalog:5432/iceberg_catalog
CATALOG_JDBC_USER: icebergcatalog
CATALOG_JDBC_PASSWORD: icebergcatalog
CATALOG_JDBC_DRIVER: org.postgresql.Driverbut apparently the postgres jdbc drivers aren't part of the image?
No suitable driver found for jdbc:postgres:/iceberg-postgres-catalog:5432/iceberg_catalog
should I just extend my own docker image and install the drivers?
Received an UnknownHostException when switching tabulario/iceberg-rest from 0.2.0 to 0.3.0
environments：
tabulario/iceberg-rest:0.3.0
flink 1.16.1
iceberg 1.2.0
When I execute the following statement in Flink SQL client：
CREATE CATALOG rest_catalog WITH (
'type'='iceberg',
'catalog-type'='rest',
'uri'='http://localhost:8181'
);
USE CATALOG rest_catalog;
CREATE TABLE all_users_sink (
database_name STRING,
table_name STRING,
`id` DECIMAL(20, 0) NOT NULL,
name STRING,
address STRING,
phone_number STRING,
email STRING,
PRIMARY KEY (database_name, table_name, `id`) NOT ENFORCED
);
An error occurred：
[ERROR] Could not execute SQL statement. Reason: org.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
The same configuration in tabulario/iceberg-rest:0.2.0, Execute fine.
Error while running
Hi team,
while trying to execute the Jar. Getting below issue:
java -jar ./build/libs/iceberg-rest-image.jar
Error: Unable to initialize main class org.apache.iceberg.rest.RESTCatalogServer
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
Could you please help me to resolve it.
Thanks!!
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.