Enigma Coding Challege
This repo is my proposed solutions to [Enigma] (http://enigma.io/) coding challenges for their data engineering position.
There are two parts to this:
Part 1. CSV manipulation:
Problems:
- String cleaning - The bio field contains text with arbitrary padding, spacing and line breaks. Normalize these values to a space-delimited string.
- Code swap - There is a supplementary CSV file for download here: state_abbreviations.csv. This "data dictionary" contains state abbreviations alongside state names. For the state field of the input CSV, replace each state abbreviation with its associated state name from the data dictionary.
- Date offset (bonus) - The start_date field contains data in a variety of formats. These may include e.g., "June 23, 1912" or "5/11/1930" (month, day, year). But not all values are valid dates. Invalid dates may include e.g., "June 2018", "3/06" (incomplete dates) or even arbitrary natural language. Add a start_date_description field adjacent to the start_date column to filter invalid date values into. Normalize all valid date values in start_date to ISO 8601 (i.e., YYYY-MM-DD).
My proposed solutions
cleaning.py
this file cleans the bio field in the csv filetest.csv
and produces output fileoutput.csv
swap.py
this file repaces abbreviations with the appropriate state names and produces output filefixed_state.csv
bonus.py
this file filters uses the custom validation class I wrotevalidate.py
to validate and normalize date strings. It also filters bad balues intostart_date_description
to the output filebonus.csv
Part 2. Scraping:
Problem:
- Write a script to scrape a [sample site] (http://data-interview.enigmalabs.org/companies/) and output its data in JSON
Proposed Solution:
scraper.py
scrapes the site usingBeautiful soup
andrequests
, and outputs its results tosolution.json