-
Get data from API end point in CSV format, and store the same in a file: -- while storing in CSV file, please add a column processed_time, which is current time at the point of scraping the data. end point: http://www.tennet.org/english/operational_management/export_data.aspx?exporttype=bidpriceladder&format=csv&datefrom=2020-01-01&dateto=2020-01-02&submit=1
-
Perform pivot using pandas on same dataframe: --where : 'PTE' as columns 'Date' as index 'total_plus' as value -- Use aggregate function if necessary while performing above operation.
-
Store data into sql database, create a database (test) and table name (test table).
-
Once stored, please perform CRUD operations on the data in database using python/java code.
Step1 : Run main.py file
Note: pivoted_data.csv has been stored as reference for final ouptut