privatekit / safepaths-datascience

Repository with various data science projects in the branches (More coming soon)

safepaths-datascience

Repository with various data science projects in the branches (More coming soon)

Data files

Links to several repositories of publicly available data are stored in the data_sources.json with the following structure:

data_sources.json:
    |--Global:
        |--source:link
    |--Nations:
        |--Regions:
            |--Provinces:
                |--source:link

Ingesting algorithms

Ingesting pipelines need to check:

• file format: csv, json, etc.
• based on the file format, select the longitude, latitude and timestamp values (where present) (in progress)
• logging errors (in progress)
• build a harmonized file containing only those three columns
• harmonize values (need to avoid doubling of data qhen present from different sources)

safepaths-datascience's Issues

Date format

At the moment the data ingestion script do not ingest timestamps or similar. The problem is that public datasets will have different formats.

1. We need something that recognizes the date/timestamp format (without loading heavy packages as pandas, if I understand well) in each dataset we ingest
2. What format we implement eventually? POSIX, I imagine.
3. This will be fundamental to periodically clear the data every 15 days

Necessity to implement logging

The "check" functions in the utils.py should log their status (both positive and errors)

Google doc

At the moment utils.py only recognizes .csv and .json. While I think those will account for most dataset, we might want to implement the Google API calls for google spreadsheets as well.

Harmonization of records

How do we make sure that public data do not overlap and numbers are summed up?

To be taken into consideration: the public data I am currently pulling do not have mobile GPS data, but refer to reference pinpoint coordinates (i.e. one central point for each state/province, or the coordinates of the hospitals).