leesf / hudi-demos

汇总Apache Hudi中的一些Demo，便于快速上手Apache Hudi(Apache Hudi Demos to help beginners know about Hudi)

Hi there 👋

About Me 😄

• I am PMC member and committer at Apache Hudi.
• I am currently using Apache Hudi to build Lakehouse at large scale.
• I am interested in big data technologies.
• I love sharing technology about Apache Hudi, please checkout the summary resources at https://github.com/leesf/hudi-resources and you can follow on belowing ApacheHudi WeChat account to get info timely.

Get in Touch 📫

• 📮 E-mail: [email protected]

hudi-demos's Issues

add mor table multiversion demo

refactor project structure to introduce hudi-demo-common

运行oss的demo时遇到问题

你好，我在运行oss 的 demo时遇到了一些问题，运行一段时间后会抛出一个异常，并且程序会终止运行。

根据异常能判断出是oss的endpoint配置的有问题，但是我以各种方式配置了很多次都没有解决，能否帮忙解决一下这个异常？我会将异常信息以及我的相关配置贴出来。

以下是我对endpoint做的配置：

    <property>
        <name>fs.oss.endpoint</name>
        <value>oss-endpoint-address</value>
        <description>oss-cn-beijing.aliyuncs.com</description>
<!--        <description>oss-cn-beijing</description>-->
<!--        <description>beijing</description>-->
    </property>

以下是异常信息：
Exception in thread "main" com.aliyun.oss.ClientException: oss-endpoint-address: nodename nor servname provided, or not known
[ErrorCode]: UnknownHost
[RequestId]: Unknown
at com.aliyun.oss.common.utils.ExceptionFactory.createNetworkException(ExceptionFactory.java:71)
at com.aliyun.oss.common.comm.DefaultServiceClient.sendRequestCore(DefaultServiceClient.java:137)
at com.aliyun.oss.common.comm.ServiceClient.sendRequestImpl(ServiceClient.java:133)
at com.aliyun.oss.common.comm.ServiceClient.sendRequest(ServiceClient.java:70)
at com.aliyun.oss.internal.OSSOperation.send(OSSOperation.java:83)
at com.aliyun.oss.internal.OSSOperation.doOperation(OSSOperation.java:145)
at com.aliyun.oss.internal.OSSObjectOperation.getObjectMetadata(OSSObjectOperation.java:496)
at com.aliyun.oss.OSSClient.getObjectMetadata(OSSClient.java:599)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystemStore.getObjectMetadata(AliyunOSSFileSystemStore.java:266)
at org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:267)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1734)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:99)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:125)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
at com.leesf.HudiOssDemo.main(HudiOssDemo.java:41)
Caused by: java.net.UnknownHostException: oss-endpoint-address: nodename nor servname provided, or not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at com.aliyun.oss.common.comm.DefaultServiceClient.sendRequestCore(DefaultServiceClient.java:134)
... 33 more
638358 [pool-7-thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from shutdown hook

大神，啥时候更新下这个demo不

bro, do you have time to update this demos, plz plz plz