rohanghorpade / programming-exercises

Set Up

1. maven needs to be set up on the local machine
2. pyspark version 3.0.0 is required

Goal

The goal of the programming exercise is

1. Read data from a json file without inferring schema.
2. Explode arrays
3. Unwrap nested structures.
4. Write the dataframe to csv
5. Create delta table
6. Write to table
7. Read from table

Additional questions

1. Spark cluster components and deployment modes
2. Caching - cache(), persist(), unpersist(), and storage levels
3. Partitioning
   1. Initial DataFrame partitioning when reading from data source
   2. Repartitioning via coalesce() vs repartition()
   3. Controlling number of shuffle partitions
4. Performance
   1. Catalyst optimizer
   2. Identifying performance bottlenecks in Spark applications
5. Transformations, actions, and other operations
   1. Wide vs Narrow
6. Joins
   1. Broadcast Joins
   2. Cross Joins
7. Defining and using User Defined Functions (UDFs)
8. Window functions
9. Streaming
   1. Checkpoints
   2. Aggregation using time windows
   3. Watermarking