olehonyshchak / wikiimagerecommendation Goto Github PK

The description is not captured in some cases, specifically:

• when the file is protected and thus "local-copied" in Wikipedia, such as Flag of the United Kingdom.svg
• sometimes description field is missing from Commons but there is other fields such as "title", "notes", for example as in Claire Clairmont, by Amelia Curran.jpg

It seems that description was converted into an incorrect format since there are some artefacts. For example, Ice House.jpg image description contains \xa0 substrings

Make some extensive data exploration to find possible problems with data collection

• Multiple Image template, such as in Transandinomys article
• Automatic taxobox template, such as in Transandinomys article
• Infobox templates such as in Barack Obama article

In some cases, parser leaves artefacts, for example:

• {{rp at the end of file [[File:Young Diffraction.png|thumb|left|[[Thomas Young (scientist)|Thomas Young]]'s [[double-slit experiment]] in 1801 showed that light can act as a [[wave]], helping to invalidate early [[elementary particle|particle]] theories of light.<ref name=Halliday/>{{rp|964}}]] from Photon page

Some images might have alternative text (alt tag) specified additionally to the caption, which is displayed when image cannot be rendered. Offen it is different from caption itself, thus it is useful to collect it as well.

Tasks

• remove file extensions from parsed titles
• merge specified files

Good to Have

• generate "processed titles" for good articles dataset

There is a problem that firstly we retrieve all main data and only after that we start the second iteration for e.g. parsing titles. And that is bad because before we start parsing captions article might have been modified already. So we should either merge all of them into the same pipeline or parse captions and headings only based on already cached data.

Originally we removed all SVG images from dataset since there were mostly auxiliary Wikipedia icons, presented on a lot of articles and causing the noise. Although, filtering them out based on SVG format is incorrect since there are a lot of legit images with that extension as well.

We need to figure out the way to remove icons without sacrificing meaningful SVG images