- Set up spark scala, run the code on terminal
- Get the desire output
- Run a test in form of a sanity test
- Generate a csv report of the desired output
Clone this repository in your computer using git clone https://github.com/profbiyi/dsti_spark_project2021.git
- Download Spark here. Put Spark command in your PATH. (I opted for Spark 3.1.2)
- On windows, download hadoop binary here and save to C:\Windows\System32. I was unable to save to CSV until this was done
- on windows download winutils for the version of hadoop here and save in HADOOP_HOME/bin
Go to https://www.kaggle.com/olistbr/brazilian-ecommerce and download the zip file.
Then unzip the archive.zip
unzip archive.zip
There are two ways to run this code.
spark-shell -i code.scala
Open a terminal and and change directory to where the unzip file are: Then run the following command:
spark-shell
- order_purchase_timestamp is in Sao Palo in Brazil, so it has to be converted to UTC
- order_delivered_customer_date assumes that all customers are in brazi so it has to be converted to UTC
- A general timezone of UTC -3 is used as the timezone for all the orders.
Run the following code:
import org.apache.spark.sql.DataFrame
def extractDeliveriesMoreThan10Days: DataFrame = {
val path = "C:/Users/ibnqu/Desktop/spark_scala_stuff/olist"
val ordersPath = s"$path/olist_orders_dataset.csv"
val ordersDF = spark.read.option("header", "true").csv(ordersPath)
ordersDF.createOrReplaceTempView("orders")
spark.sql("select datediff(to_utc_timestamp(order_delivered_customer_date, 'UTC-3'), to_utc_timestamp(order_purchase_timestamp, 'UTC-3')) as delivery_delay_in_days from orders where datediff(to_utc_timestamp(order_delivered_customer_date, 'UTC-3'), to_utc_timestamp(order_purchase_timestamp, 'UTC-3')) > 10 ")
}
deliveries of more than 10 days
extractDeliveriesMoreThan10Days.show
verify we have no delivery of less than 10 days, should return true:
extractDeliveriesMoreThan10Days.filter(col("delivery_delay_in_days") <= 10).show
extractDeliveriesMoreThan10Days.filter(col("delivery_delay_in_days") <= 10).count == 0
extractDeliveriesMoreThan10Days.coalesce(1).write.option("header", "true").csv(""delivery_delay_in_days.csv")
Ahmed Oladapo, mail: [email protected]