takluyver / spotting-stat-claims

Fullfact hackathon - identifying statistical claims

Spotting statistical claims

Full Fact Hackathon at PyData London 2017

We are scanning text to identify statistical claims. We look for numbers, monetary amounts, percentages, and terms such as 'increase' or 'decrease'. We try to avoid false positives, such as "Article 37".

This works with XML files from this Hansard data set.

To run the demo:

./statclaims.py --html hansard/src/debates2017-01-23d.xml

Then open index.html to see the output. You can pass multiple files at once. Leaving out --html will show output in the terminal.

Notes in a Google Doc.

spotting-stat-claims's Issues

Present sentences

Rather than snippets.

Classify matches?

Would it be useful to classify results (e.g. monetary values), so these can be selected out easily?

Combine consecutive matches

At present, we match each recognised part separately, e.g.

• "20,000" -> ("20", "000")
• "£10 billion" -> ("£", "10", "billion").

Ideally, we'd like one match for each of those.

Mistaken matches

Most of the matches look useful, but here are a few that we may be able to improve.

Matched words in bold.

• "would be nothing less than a betrayal"
• "trends in fire and rescue response times."
• "on page 90"
• "investment in HS2"
• "Universal coverage with 3G and 4G"

For the last two, drop tokens which include both letters and numbers?

• "New clause 1 of the"