Potential Data Cleanup activities about datasets

[copiousfreetime]

Doing a few more text checks, and it looks like there are a few more fields that have leading / trailing whitespace. I figure I'll just throw other potential cleanup items on here.

• unsplash_photos.exif_camera_make
• unsplash_photos.exif_camera_model
• unsplash_photos.photo_location_name
• unsplash_conversions.keyword
• unsplash_keywords.keyword

As an example:

unsplash_lite=# select '>' || keyword || '<', count(*) from unsplash_keywords where keyword IN (' wallpaper', 'wallpaper', 'wallpaper ') group by 1 order by 2 desc;
   ?column?   | count
--------------+-------
 >wallpaper<  |  1951
 > wallpaper< |    12
 >wallpaper < |     5

[TimmyCarbone]

Hey! Thank you for all this feedback!

We haven't fully normalized these location fields internally yet, that's why they're still in "free form" (we do offer Google Maps suggestions in the form but sometimes they're not being used). There are multiple challenges here (including i18n) and I don't think this is a priority for us in the near future. I would say that, in this case and for now, it'd be up to the person utilizing the dataset to do some cleaning if they want better data quality on specific fields.

The conversion_country is a guaranteed ISO 3166-1 code (2 letter country code). The lookup is based upon the GeoLite2-City database from Maxmind.

Thank you, hopefully we can improve the dataset over time thank to everyone's feedback.
Also, of course, whenever we're implementing new things internally, it'll be reflected on the datasets.

[copiousfreetime]

@TimmyCarbone no worries, just wanting to make sure to utilize this dataset correctly. And I'll plan on doing various cleanup activities 😄

[TimmyCarbone]

Let's keep this issue opened so it stays on our and everyones radar though :)

I believe that having clean fields is important. We might and will probably get to normalizing the location fields at some point in the future, it's just not really planned yet. I'll keep you and everyone posted on this issue as soon as we have a plan to tackle it.

Also, if anyone wants to give it a shot, we'd be happy to implement good solutions from the open-source community!

[TimmyCarbone]

For now, we'll address the trailing/leading whitespace issue on all the fields you mentioned.
These issues will be fixed in the 1.1.0 release.

We'll keep the normalization feature aside for now since it's a much bigger project that requires more resources than we have at the moment.