Thanks to @anarasimham for the python skeleton that was modifed for this data generator.
https://github.com/anarasimham/data-gen
The data generater uses the faker python library to generate random values for each attribute in a given schema. The ouput of the script is json format. In the script there are 5 unique data generators:
- CustomerDataGenerator
- SalesRepDataGenerator
- POSDataGenerator
- POSCustomerDataGenerator
- ManufacturingDataGenerator
The default data genenerator in the script is the POSDataGenerator.This can be modifed by changed the selected class used for generating the data.
To modifiy what data generator is used, modifiy the d variable to point to one of the 5 unique data generators listed above. To modify the number of records to produce, modify the rage defined in the for loop. In the example below, the data generator will product 20,000 records before terminating.
if __name__ == '__main__':
d = POSDataGenerator()
for i in range(1,20000):
record = d.gen_row()
record = json.dumps(record)
producer.send(KAFKA_TOPIC, record)
print(record)
A kafka producer has been included as well as a sleep timer that will regulate the data generation. The timer is defaulted to 1 json record per second but this can be adjusted as required.
random, json, time, datetime, faker kafka
nohup python data-generator-kafka-producer.py &
python data-generator-kafka-producer.py