simonellistonball / indoorsparklers

A fake data utility for Spark

Indoor Sparklers

Generates random data of a specific template and form using Spark, outputting to ORC files and creating a table in Hive based on said ORC files.

Everyone knows that indoor fireworks are not real. This program is the beginnings of a utility to generate fake data in the Spark job, mainly for unit testing, and load testing Spark jobs. It's kind of basic for now, but will hopefully grow as I end up doing a lot more Spark work.

• TODO: configurable data creation pattern
• TODO: other output formats from command line config
• TODO: parameter sanity checking
• TODO: auto-guess-magic parallelism settings

indoorsparklers's Issues

It is currently hard to judge the right parallelism settings for the job

The parellelism and other resource could be calculated instead of relying on the user to tune them on the spark-submit command line

(cores, executors, partition count, memory sizes)

Cannot change the schema

It would be good to provide a schema on the command line, or through some sort of template or configuration file to then generate the data.

Parameters should be checked

The number of records must be a multiple of the number of partitions, since the code doesn't pickup spares.