how to query on SPARKSQL about struct-type-encoder

[BenFradet]

hello, it's not clear to me where you're using the library?

[heenasalim]

package Heena

import org.apache.spark.SparkContext

import org.apache.spark.sql._
import org.apache.spark.sql.Row
import org.apache.spark.sql.functions
import org.apache.spark.sql.types;
import org.apache.spark.sql.SparkSession

object vendor {

def main(args:Array[String])
{

//Reading the files in dataframe forms
//The spark.read.csv function is dataframe
//when we pass case class/type to this data frame like sparkread.csv.as[Person]/csv.as[String]
//it becomes dataset which can be handle exacty as method 1 project1 ie select and filter

 val sparksession = SparkSession.builder().master("local").
 appName("kmart_vendor_pakage_location").getOrCreate();
 import sparksession.implicits._

var work__store_level_vend_pack_loc_final_data =
  
  sparksession.read.format("csv")
 .option("header", "true")
 .option("delimiter", "|")
 .option("inferSchema", "true")
 .load("C:\\Users\\jabin\\Desktop\\project_files\\work__store_level_vend_pack_loc_final_data.txt");
 
  work__store_level_vend_pack_loc_final_data.registerTempTable("work__store_level_vend_pack_loc_final_data_table");
 
var r1 = sparksession.sqlContext.sql( "SELECT shc_item_id ,'K' as source_owner_cd,item_purchase_status_cd, vendor_package_id,vendor_package_purchase_status_cd,flow_type_cd as vendor_package_flow_type_cd,vendor_carton_qty,vendor_stock_nbr,ksn_package_id,ksn_purchase_status_cd,import_ind,sears_divission_nbr,sears_item_nbr,sears_sku_nbr,scan_based_trading_ind,cross_merchandising_cd,retail_carton_vendor_package_id,vendor_package_owner_cd,can_carry_model_id,'' AS days_to_check_begin_day_qty,'' AS days_to_check_end_day_qty ,dotcom_allocation_ind ,retail_carton_internal_package_qty,allocation_replenishment_cd,shc_item_type_cd,idrp_order_method_cd,source_package_qty as store_source_package_qty,order_duns_nbr FROM work__store_level_vend_pack_loc_final_data_table WHERE flow_type_cd = 'JIT'  OR servicing_dc_nbr > '0' ")
 // }}

The whole spark scala code.

My problem is I want to select distinct columns from r1 , for that I am using following
sparksession.sqlContext.sql("select distinct * from work__store_level_vend_pack_loc_final_data_table")
.collect.foreach(println);
but it showing result on original table not on latest r1.

[heenasalim]

@BenFradet
work__store_level_vend_pack_loc_final_data.txt

input files are attached.

[BenFradet]

Hello, it doesn't seem that you use this library, I would suggest reaching out to the spark community as a whole: https://spark.apache.org/community.html

[heenasalim]

What s relation between spark community and github? there is noneed of the tag apache-spark as no any tag field is there in Github.

…

On Wed, Mar 27, 2019 at 8:58 PM Ben Fradet ***@***.***> wrote: Closed #41 <#41>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#41 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AjT08M8AqQ2ITXhEv-K8XblR6yWkwvSWks5va44hgaJpZM4cN6qn> .

[BenFradet]

if you're looking for the spark github repo it's there: https://github.com/apache/spark