tabular-io / iceberg-rest-image Goto Github PK
View Code? Open in Web Editor NEWSimple project to expose a catalog over REST using a Java catalog backend
License: Apache License 2.0
Simple project to expose a catalog over REST using a Java catalog backend
License: Apache License 2.0
Is there any possibility of having a health check for use in Kubernetes? We've seen that when using iceberg-rest
with PostgreSQL if the connection to the database server drops, the iceberg-rest
server needs rebooting. This could be easily handled automatically from Kubernetes by having a health check but I don't know how much effort it would be to add in health checks for each catalog (where necessary)
Setting CATALOG_IO__IMPL=org.apache.iceberg.nessie.NessieCatalog
throws an exception immediately at startup:
2023-01-18T15:54:22.583 INFO [org.apache.iceberg.CatalogUtil] - Loading custom FileIO implementation: org.apache.iceberg.nessie.NessieCatalog
Exception in thread "main" java.lang.IllegalArgumentException: Cannot initialize FileIO, missing no-arg constructor: org.apache.iceberg.nessie.NessieCatalog
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:311)
at org.apache.iceberg.jdbc.JdbcCatalog.initialize(JdbcCatalog.java:101)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:238)
at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:283)
at org.apache.iceberg.rest.RESTCatalogServer.backendCatalog(RESTCatalogServer.java:83)
at org.apache.iceberg.rest.RESTCatalogServer.main(RESTCatalogServer.java:87)
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.io.FileIO
Missing org.apache.iceberg.nessie.NessieCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.nessie.NessieCatalog]
at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250)
at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32)
at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220)
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:308)
... 5 more
Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.nessie.NessieCatalog
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:467)
at org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149)
at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:307)
... 5 more
Hi. I'm attempting to use a MySQL database instead of SQLite, which is the default for the catalog database. However, it appears that there is no MySQL driver class available, as shown below in the Apache Iceberg documentation:
Class.forName("com.mysql.cj.jdbc.Driver"); // ensure JDBC driver is at runtime classpath
Map<String, String> properties = new HashMap<>();
properties.put(CatalogProperties.CATALOG_IMPL, JdbcCatalog.class.getName());
properties.put(CatalogProperties.URI, "jdbc:mysql://localhost:3306/test");
properties.put(JdbcCatalog.PROPERTY_PREFIX + "user", "admin");
properties.put(JdbcCatalog.PROPERTY_PREFIX + "password", "pass");
properties.put(CatalogProperties.WAREHOUSE_LOCATION, "s3://warehouse/path");
Configuration hadoopConf = new Configuration(); // configs if you use HadoopFileIO
JdbcCatalog catalog = CatalogUtil.buildIcebergCatalog("test_jdbc_catalog", properties, hadoopConf);
When I changed the environment like this: "CATALOG_URI: "jdbc:mysql://10.10.50.182/lakehouse", the logs output is as follows:
Exception in thread "main" org.apache.iceberg.jdbc.UncheckedSQLException: Failed to connect:
jdbc:mysql://10.10.50.182/lakehouse
at org.apache.iceberg.jdbc.JdbcClientPool.newClient(JdbcClientPool.java:57)
at org.apache.iceberg.jdbc.JdbcClientPool.newClient(JdbcClientPool.java:30)
at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:125)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.jdbc.JdbcCatalog.initializeCatalogTables(JdbcCatalog.java:155)
at org.apache.iceberg.jdbc.JdbcCatalog.initialize(JdbcCatalog.java:141)
at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:255)
at org.apache.iceberg.CatalogUtil.buildIcebergCatalog(CatalogUtil.java:309)
at org.apache.iceberg.rest.RESTCatalogServer.backendCatalog(RESTCatalogServer.java:86)
at org.apache.iceberg.rest.RESTCatalogServer.main(RESTCatalogServer.java:90)
Caused by: java.sql.SQLException: No suitable driver found for jdbc:mysql://10.10.50.182/lakehouse
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:706)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
at org.apache.iceberg.jdbc.JdbcClientPool.newClient(JdbcClientPool.java:55)
... 10 more
Do you have any plans to address this issue? Thank you.
From trino, I can create the schema/namespace. On CREATE TABLE, writing the underlying files to s3 (actually, minio) works, but on committing the metadata, thows an exception:
2023-01-18T15:47:10.326 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] - Error processing REST request
org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=500, type=RuntimeIOException, message=Failed to get file system for path: s3://testdata/tinynation/metadata/00000-00b584bb-e702-4484-9968-92c0948f08ca.metadata.json)
org.apache.iceberg.exceptions.RuntimeIOException: Failed to get file system for path: s3://testdata/tinynation/metadata/00000-00b584bb-e702-4484-9968-92c0948f08ca.metadata.json
at org.apache.iceberg.hadoop.Util.getFs(Util.java:54)
at org.apache.iceberg.hadoop.HadoopOutputFile.fromPath(HadoopOutputFile.java:53)
at org.apache.iceberg.hadoop.HadoopFileIO.newOutputFile(HadoopFileIO.java:83)
at org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:159)
at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:105)
at org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:135)
at org.apache.iceberg.rest.CatalogHandlers.create(CatalogHandlers.java:315)
at org.apache.iceberg.rest.CatalogHandlers.updateTable(CatalogHandlers.java:263)
at org.apache.iceberg.rest.RESTCatalogAdapter.handleRequest(RESTCatalogAdapter.java:343)
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:384)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.iceberg.hadoop.Util.getFs(Util.java:52)
... 42 more
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:401)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:713)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:833)
I think this is a config issue, but I haven't been able to suss out what env vars I need to declare.
I want to add iceberg REST catalog with the value of OAUTH2
for the option of iceberg.rest-catalog.security
to my trino cluster, for which REST catalog server needs to be run with support of OAuth2 authentication.
I have added some codes at the end of RESTCatalogServer
like this.
RESTCatalogAdapter adapter = new RESTCatalogAdapter(backendCatalog());
RESTCatalogServlet servlet = new RESTCatalogServlet(adapter);
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.NO_SESSIONS);
context.setContextPath("/");
ServletHolder servletHolder = new ServletHolder(servlet);
servletHolder.setInitParameter("javax.ws.rs.Application", "ServiceListPublic");
context.addServlet(servletHolder, "/*");
context.setVirtualHosts(null);
context.setGzipHandler(new GzipHandler());
Server httpServer =
new Server(PropertyUtil.propertyAsInt(System.getenv(), "REST_PORT", 8181));
httpServer.setHandler(context);
httpServer.start();
// ADDED CODES!!!!!!!
SessionCatalog.SessionContext sessionContext =
new SessionCatalog.SessionContext(
UUID.randomUUID().toString(),
"user",
ImmutableMap.of("credential", "user:secret"),
ImmutableMap.of());
RESTCatalog catalog = new RESTCatalog(sessionContext, (config) -> adapter);
catalog.initialize(
"prod", ImmutableMap.of(CatalogProperties.URI, "ignored", "token", "bearer-token"));
But it seems, this is not OAuth2 enabled server.
Anybody knows how to add OAuth2 authentication to RESTCatalogServer
?
Is there a way to connect to a Hive Metastore Catalog for this rest interface?
if not, will it be added it in the future?
Thanks!
hi, is there a well-tested way to deploy REST Catalog on kubernetes? great thanks.
The documentation of this image on DockerHub mentions that it uses an 'in memory backend'. However, I tested it locally and noticed that the catalog was in fact persisted (i.e. it still existed after rebooting the container). I assume this is due to an error in the source code.
I think the connection string jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory
in
catalogProperties.putIfAbsent(
CatalogProperties.URI, "jdbc:sqlite:file:/tmp/iceberg_rest_mode=memory");
needs to be replaced. Currently, this creates an iceberg_rest_mode=memory
file in the tmp
directory of the container that is persisted and is just a regular sqlite
DB file.
Hello,
I was using the example project docker-spark-iceberg in a Docker environment and I've to replace the org.apache.iceberg.aws.s3.S3FileIO
with org.apache.iceberg.hadoop.HadoopFileIO
for legacy problems with s3a
schema.
I've the following error:
22/11/17 13:36:37 ERROR Utils: Aborting task
org.apache.iceberg.exceptions.ServiceFailureException: Server error: RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
at org.apache.iceberg.rest.ErrorHandlers.lambda$defaultErrorHandler$3(ErrorHandlers.java:125)
at java.base/java.util.function.Consumer.lambda$andThen$0(Consumer.java:65)
at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:129)
at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:193)
at org.apache.iceberg.rest.HTTPClient.post(HTTPClient.java:228)
at org.apache.iceberg.rest.RESTClient.post(RESTClient.java:68)
at org.apache.iceberg.rest.RESTTableOperations.commit(RESTTableOperations.java:130)
at org.apache.iceberg.BaseTransaction.commitCreateTransaction(BaseTransaction.java:277)
at org.apache.iceberg.BaseTransaction.commitTransaction(BaseTransaction.java:256)
at org.apache.iceberg.spark.source.StagedSparkTable.commitStagedChanges(StagedSparkTable.java:35)
at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.$anonfun$writeToTable$1(WriteToDataSourceV2Exec.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable(WriteToDataSourceV2Exec.scala:491)
at org.apache.spark.sql.execution.datasources.v2.TableWriteExecHelper.writeToTable$(WriteToDataSourceV2Exec.scala:486)
at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.writeToTable(WriteToDataSourceV2Exec.scala:108)
at org.apache.spark.sql.execution.datasources.v2.AtomicCreateTableAsSelectExec.run(WriteToDataSourceV2Exec.scala:131)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:116)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:636)
at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:566)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.base/java.lang.Thread.run(Thread.java:829)
How can I add the hadoop-aws
(https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws) dependency to the RESTCatalog?
Thanks!
Hi,
some of your files do contain a license header but not all.
This makes it problematic to use.
Would you be willing to add a LICENSE to the repository?
Hi team,
thanks for this very useful project. I've been having good success testing various things with iceberg using this catalog.
Recently, I've been getting 500
s under somewhat heavy load:
2023-08-10T14:44:52.707 ERROR [org.apache.iceberg.rest.RESTCatalogServlet] - Error processing REST request
org.apache.iceberg.exceptions.RESTException: Unhandled error: ErrorResponse(code=500, type=UncheckedSQLException, message=Unknown failure)
I was able to find the following in the stack trace:
Caused by: org.sqlite.SQLiteException: [SQLITE_BUSY] The database file is locked (database is locked)
at org.sqlite.core.DB.newSQLException(DB.java:1179)
at org.sqlite.core.DB.newSQLException(DB.java:1190)
at org.sqlite.core.DB.execute(DB.java:985)
at org.sqlite.core.DB.executeUpdate(DB.java:1054)
at org.sqlite.jdbc3.JDBC3PreparedStatement.lambda$executeLargeUpdate$2(JDBC3PreparedStatement.java:118) iceberg-rest-s3-aws | at org.sqlite.jdbc3.JDBC3Statement.withConnectionTimeout(JDBC3Statement.java:454)
at org.sqlite.jdbc3.JDBC3PreparedStatement.executeLargeUpdate(JDBC3PreparedStatement.java:117)
at org.sqlite.jdbc3.JDBC3PreparedStatement.executeUpdate(JDBC3PreparedStatement.java:99)
at org.apache.iceberg.jdbc.JdbcTableOperations.lambda$updateTable$0(JdbcTableOperations.java:164)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.jdbc.JdbcTableOperations.updateTable(JdbcTableOperations.java:153)
at org.apache.iceberg.jdbc.JdbcTableOperations.doCommit(JdbcTableOperations.java:115)
... 43 more
at org.apache.iceberg.rest.RESTCatalogAdapter.execute(RESTCatalogAdapter.java:401)
at org.apache.iceberg.rest.RESTCatalogServlet.execute(RESTCatalogServlet.java:100)
at org.apache.iceberg.rest.RESTCatalogServlet.doPost(RESTCatalogServlet.java:78)
My guess was that there is some sort of connection pooling of JDBC connections going on and sqlite doesn't play nice with that. I checked the source of JdbcClientPool and saw that the default pool size is 2, this increased my suspicions.
I set it to 1 using CATALOG_CLIENTS=1
and haven't had these errors since.
I propose to set this to 1
if the default sqlite JDBC connection string is used. I'd be happy to draft a PR if someone else can confirm my suspicions.
Cheers,
Johannes
Security findings found via Trivy when using that need to be resolved. Unsure if this should be reported here or upstream.
In summary there are 5 medium and 4 high CVE vulnerabilities that can be resolved with updated dependencies within iceberg-rest-image-all.jar
Note that this is with the following contents in build.gradle
ext {
icebergVersion = '1.5.2'
hadoopVersion = '3.4.0'
}
Total: 39 (UNKNOWN: 0, LOW: 33, MEDIUM: 6, HIGH: 0, CRITICAL: 0)
┌──────────────────┬────────────────┬──────────┬──────────┬──────────────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├──────────────────┼────────────────┼──────────┼──────────┼──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ coreutils │ CVE-2016-2781 │ LOW │ affected │ 8.32-4.1ubuntu1.2 │ │ coreutils: Non-privileged session can escape to the parent │
│ │ │ │ │ │ │ session in chroot │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-2781 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ gcc-12-base │ CVE-2022-27943 │ │ │ 12.3.0-1ubuntu1~22.04 │ │ binutils: libiberty/rust-demangle.c in GNU GCC 11.2 allows │
│ │ │ │ │ │ │ stack exhaustion in demangle_const │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-27943 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ gpgv │ CVE-2022-3219 │ │ │ 2.2.27-3ubuntu2.1 │ │ gnupg: denial of service issue (resource consumption) using │
│ │ │ │ │ │ │ compressed packets │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-3219 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libc-bin │ CVE-2016-20013 │ │ │ 2.35-0ubuntu3.7 │ │ sha256crypt and sha512crypt through 0.6 allow attackers to │
│ │ │ │ │ │ │ cause a denial of... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-20013 │
├──────────────────┤ │ │ │ ├───────────────┤ │
│ libc6 │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libgcc-s1 │ CVE-2022-27943 │ │ │ 12.3.0-1ubuntu1~22.04 │ │ binutils: libiberty/rust-demangle.c in GNU GCC 11.2 allows │
│ │ │ │ │ │ │ stack exhaustion in demangle_const │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-27943 │
├──────────────────┼────────────────┼──────────┤ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libgcrypt20 │ CVE-2024-2236 │ MEDIUM │ │ 1.9.4-3ubuntu3 │ │ libgcrypt: vulnerable to Marvin Attack │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-2236 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libgssapi-krb5-2 │ CVE-2024-26462 │ │ │ 1.19.2-2ubuntu0.3 │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libk5crypto3 │ CVE-2024-26462 │ MEDIUM │ │ │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libkrb5-3 │ CVE-2024-26462 │ MEDIUM │ │ │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libkrb5support0 │ CVE-2024-26462 │ MEDIUM │ │ │ │ krb5: Memory leak at /krb5/src/kdc/ndr.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26462 │
│ ├────────────────┼──────────┤ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26458 │ LOW │ │ │ │ krb5: Memory leak at /krb5/src/lib/rpc/pmap_rmt.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26458 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26461 │ │ │ │ │ krb5: Memory leak at /krb5/src/lib/gssapi/krb5/k5sealv3.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26461 │
├──────────────────┼────────────────┼──────────┤ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ liblzma5 │ CVE-2020-22916 │ MEDIUM │ │ 5.2.5-2ubuntu1 │ │ Denial of service via decompression of crafted file │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2020-22916 │
├──────────────────┼────────────────┼──────────┤ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libncurses6 │ CVE-2023-45918 │ LOW │ │ 6.3-2ubuntu0.1 │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ libncursesw6 │ CVE-2023-45918 │ │ │ │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libpcre3 │ CVE-2017-11164 │ │ │ 2:8.39-13ubuntu0.22.04.1 │ │ pcre: OP_KETRMAX feature in the match function in │
│ │ │ │ │ │ │ pcre_exec.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2017-11164 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libpng16-16 │ CVE-2022-3857 │ │ │ 1.6.37-3build5 │ │ libpng: Null pointer dereference leads to segmentation fault │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-3857 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libstdc++6 │ CVE-2022-27943 │ │ │ 12.3.0-1ubuntu1~22.04 │ │ binutils: libiberty/rust-demangle.c in GNU GCC 11.2 allows │
│ │ │ │ │ │ │ stack exhaustion in demangle_const │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-27943 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libsystemd0 │ CVE-2023-7008 │ │ │ 249.11-0ubuntu3.12 │ │ systemd-resolved: Unsigned name response in signed zone is │
│ │ │ │ │ │ │ not refused when DNSSEC=yes... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-7008 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libtinfo6 │ CVE-2023-45918 │ │ │ 6.3-2ubuntu0.1 │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libudev1 │ CVE-2023-7008 │ │ │ 249.11-0ubuntu3.12 │ │ systemd-resolved: Unsigned name response in signed zone is │
│ │ │ │ │ │ │ not refused when DNSSEC=yes... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-7008 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ libzstd1 │ CVE-2022-4899 │ │ │ 1.4.8+dfsg-3build1 │ │ zstd: mysql: buffer overrun in util.c │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2022-4899 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ locales │ CVE-2016-20013 │ │ │ 2.35-0ubuntu3.7 │ │ sha256crypt and sha512crypt through 0.6 allow attackers to │
│ │ │ │ │ │ │ cause a denial of... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2016-20013 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ login │ CVE-2023-29383 │ │ │ 1:4.8.1-2ubuntu2.2 │ │ shadow: Improper input validation in shadow-utils package │
│ │ │ │ │ │ │ utility chfn │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-29383 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ ncurses-base │ CVE-2023-45918 │ │ │ 6.3-2ubuntu0.1 │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ ncurses-bin │ CVE-2023-45918 │ │ │ │ │ ncurses 6.4-20230610 has a NULL pointer dereference in │
│ │ │ │ │ │ │ tgetstr in tinf ...... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-45918 │
│ ├────────────────┤ │ │ ├───────────────┼──────────────────────────────────────────────────────────────┤
│ │ CVE-2023-50495 │ │ │ │ │ ncurses: segmentation fault via _nc_wrap_entry() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-50495 │
├──────────────────┼────────────────┤ │ ├──────────────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ passwd │ CVE-2023-29383 │ │ │ 1:4.8.1-2ubuntu2.2 │ │ shadow: Improper input validation in shadow-utils package │
│ │ │ │ │ │ │ utility chfn │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-29383 │
└──────────────────┴────────────────┴──────────┴──────────┴──────────────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘
2024-05-29T11:12:12-04:00 INFO Table result includes only package filenames. Use '--format json' option to get the full path to the package file.
Java (jar)
Total: 9 (UNKNOWN: 0, LOW: 0, MEDIUM: 5, HIGH: 4, CRITICAL: 0)
┌─────────────────────────────────────────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬────────────────────────┬──────────────────────────────────────────────────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ ch.qos.logback:logback-classic (iceberg-rest-image-all.jar) │ CVE-2023-6378 │ HIGH │ fixed │ 1.2.10 │ 1.3.12, 1.4.12, 1.2.13 │ logback: serialization vulnerability in logback receiver │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-6378 │
├─────────────────────────────────────────────────────────────┤ │ │ │ │ │ │
│ ch.qos.logback:logback-core (iceberg-rest-image-all.jar) │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┤ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ com.nimbusds:nimbus-jose-jwt (iceberg-rest-image-all.jar) │ CVE-2023-52428 │ MEDIUM │ │ 9.30.2 │ 9.37.2 │ Denial of Service in Connect2id Nimbus JOSE+JWT │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2023-52428 │
├─────────────────────────────────────────────────────────────┼────────────────┤ │ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ io.netty:netty-codec-http (iceberg-rest-image-all.jar) │ CVE-2024-29025 │ │ │ 4.1.100.Final │ 4.1.108.Final │ netty-codec-http: Allocation of Resources Without Limits or │
│ │ │ │ │ │ │ Throttling │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-29025 │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┤ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ org.apache.commons:commons-compress │ CVE-2024-25710 │ HIGH │ │ 1.24.0 │ 1.26.0 │ commons-compress: Denial of service caused by an infinite │
│ (iceberg-rest-image-all.jar) │ │ │ │ │ │ loop for a corrupted... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-25710 │
│ ├────────────────┤ │ │ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-26308 │ │ │ │ │ commons-compress: OutOfMemoryError unpacking broken Pack200 │
│ │ │ │ │ │ │ file │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-26308 │
├─────────────────────────────────────────────────────────────┼────────────────┼──────────┤ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ org.apache.commons:commons-configuration2 │ CVE-2024-29131 │ MEDIUM │ │ 2.8.0 │ 2.10.1 │ commons-configuration: StackOverflowError adding property in │
│ (iceberg-rest-image-all.jar) │ │ │ │ │ │ AbstractListDelimiterHandler.flattenIterator() │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-29131 │
│ ├────────────────┤ │ │ │ ├──────────────────────────────────────────────────────────────┤
│ │ CVE-2024-29133 │ │ │ │ │ commons-configuration: StackOverflowError calling │
│ │ │ │ │ │ │ ListDelimiterHandler.flatten(Object, int) with a cyclical │
│ │ │ │ │ │ │ object tree │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-29133 │
├─────────────────────────────────────────────────────────────┼────────────────┤ │ ├───────────────────┼────────────────────────┼──────────────────────────────────────────────────────────────┤
│ org.apache.zookeeper:zookeeper (iceberg-rest-image-all.jar) │ CVE-2024-23944 │ │ │ 3.8.3 │ 3.8.4, 3.9.2 │ Information disclosure in persistent watchers handling in │
│ │ │ │ │ │ │ Apache ZooKe ... │
│ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2024-23944 │
└─────────────────────────────────────────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴────────────────────────┴──────────────────────────────────────────────────────────────┘
Hi guys,
Are you going to add GCS support?
Any ETA?
Im running this docker compose file:
---
version: "3"
services:
iceberg-postgres-catalog:
build:
context: .
image: postgres:latest
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_USER: icebergcatalog
POSTGRES_PASSWORD: icebergcatalog
POSTGRES_DB: iceberg_catalog
iceberg-rest-catalog:
image: tabulario/iceberg-rest:latest
ports:
- "8181:8181"
environment:
CATALOG_URI: jdbc:postgres:/iceberg-postgres-catalog:5432/iceberg_catalog
CATALOG_JDBC_USER: icebergcatalog
CATALOG_JDBC_PASSWORD: icebergcatalog
CATALOG_JDBC_DRIVER: org.postgresql.Driver
but apparently the postgres jdbc drivers aren't part of the image?
No suitable driver found for jdbc:postgres:/iceberg-postgres-catalog:5432/iceberg_catalog
should I just extend my own docker image and install the drivers?
environments:
tabulario/iceberg-rest:0.3.0
flink 1.16.1
iceberg 1.2.0
When I execute the following statement in Flink SQL client:
CREATE CATALOG rest_catalog WITH (
'type'='iceberg',
'catalog-type'='rest',
'uri'='http://localhost:8181'
);
USE CATALOG rest_catalog;
CREATE TABLE all_users_sink (
database_name STRING,
table_name STRING,
`id` DECIMAL(20, 0) NOT NULL,
name STRING,
address STRING,
phone_number STRING,
email STRING,
PRIMARY KEY (database_name, table_name, `id`) NOT ENFORCED
);
An error occurred:
[ERROR] Could not execute SQL statement. Reason: org.apache.iceberg.exceptions.ServiceFailureException: Server error: SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.
The same configuration in tabulario/iceberg-rest:0.2.0, Execute fine.
Hi team,
while trying to execute the Jar. Getting below issue:
java -jar ./build/libs/iceberg-rest-image.jar
Error: Unable to initialize main class org.apache.iceberg.rest.RESTCatalogServer
Caused by: java.lang.NoClassDefFoundError: javax/servlet/Servlet
Could you please help me to resolve it.
Thanks!!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.