andersonfrailey / cpstaxunits Goto Github PK

In looking to import the 2015 CPS ASEC, I saw cpsmar_2015.py here, which seems to be the only Python script out there for importing this file. I think this can be sped up by reading the file as a Pandas Series and then extracting the fields with string functions, then merging the person/family/household features by their common identifiers, rather than one record at a time.

I gave this a starting shot in this notebook, starting with the person fields only, e.g.:

dats = pd.read_csv('http://thedataweb.rm.census.gov/pub/cps/march/' +
                   'asec2015_pubuse.dat.gz', compression='gzip', 
                   header=0).iloc[:, 0]
rec_type = dats.str[0]
p = dats[rec_type == '3']
df_p = pd.DataFrame()
df_p['precord'] = p.str[0]
df_p['ph_seq'] = p.str[1:6]
# And so on...

This takes about 5 minutes to run, so I'd expect under 15 minutes with family and household records, and merging them (compared to 2 hours as-is).

Curious what you think--I'm happy to see it through and ensure it matches the current script. At some point it might even be cool to build a Python version of the ipumsr R package (GitHub) which parses the data dictionary to avoid hard-coding string positions.