city-bureau / city-scrapers-core Goto Github PK

Opening an issue for my own investigation. We've been having issues where scrapers that are in a repository (Fresno & Atlanta have both seen it happen) are not showing up in the feed with no errors of any kind being written to the CI logs.

I traced down one problem where a missing "location.name" field on a scraper was causing JSON validation to fail, but for some reason that failure wasn't logged or propagated.

I assume the other failures we're currently investigating are a similar kind of silent failure.

on the cle_city_council.py legistar spider, the site has a missing time for a meeting entry (see screenshot), this missing time was messing up the _get_status() method in spider.py, ultimately breaking the spider. tbh, i'm not 100% sure best way to handle this bug. this spider is a legistar spider and i know legistar spiders have their own template and such.

@rajivsinclair

PR is here:
#20

Create an extension that writes scraper results including the number of meetings scraped to a JSON file, and updates an HTML page with the status of each scraper as well as a basic chart showing their results over time

i'm creating a new ticket in core to track the progress of creating a mixin/subclass for all websites that use the Events Calendar plugin for WordPress, which has a REST API.

some currently affected spiders:

City-Bureau/city-scrapers-atl#5
City-Bureau/city-scrapers-det#22
City-Bureau/city-scrapers-det#23
City-Bureau/city-scrapers-fresno#23

it looks like this sort of thing has been implemented in two different ways:

• using mixins like in detroit
• directly subclassing CityScrapersSpider like with LegistarSpider

is there a good reason to prefer either implementation?

Summary

Issue appeared here: https://github.com/City-Bureau/city-scrapers-atl/runs/7603497825?check_suite_focus=true

CI Test failing due to the following error

  File "/home/runner/work/city-scrapers-atl/city-scrapers-atl/.venv/bin/scrapy", line 8, in <module>
    sys.exit(execute())
  File "/home/runner/work/city-scrapers-atl/city-scrapers-atl/.venv/lib/python3.9/site-packages/scrapy/cmdline.py", line 149, in execute
    cmd.add_options(parser)
  File "/home/runner/work/city-scrapers-atl/city-scrapers-atl/.venv/lib/python3.9/site-packages/city_scrapers_core/commands/validate.py", line 23, in add_options
    parser.add_option(
AttributeError: 'ScrapyArgumentParser' object has no attribute 'add_option'
Error: Process completed with exit code 123.

spider is using parser.add_option (which isn't an available method for the underlying object) instead of parser.add_argument.

Proposed fix

Just update the Command.add_options method to call parser.add_argument instead of parser.add_option.

• NOTE: I haven't looked at where in production the validate --all command is being run, but the above fix should fix the error assuming that's the intended functionality

Status

PR incoming shortly

The current Legistar package has worked great, but it's tricky to install and goes outside of Scrapy for functionality. It may be more straightforward to create our own based on it that only uses the functionality we need and is built on Scrapy's requests