dylarm / parse_landsat_csv Goto Github PK

Is your feature request related to a problem? Please describe.
The program takes forever to load the ARD_TILES.csv file.

Describe the solution you'd like
Either a progress bar or some sort of iterative method of loading.

Describe alternatives you've considered

1. Find a way of having pandas tell you the progress of loading the file. Maybe by feeding it in through StringIO?
2. Don't use pandas. Instead, just process each line on its own. The CSV file is relatively well-formatted.

Is your feature request related to a problem? Please describe.
USGS provides the metadata file as a gz-compressed CSV file. Having the extra step of decompressing it wastes disk space.

Describe the solution you'd like
Load the file as-is.

Describe alternatives you've considered

1. Some sort of JIT decompression? Decompress only in memory.
2. Pandas has native compression support for CSV files.

Additional context
May be difficult to do if also doing #1