GithubHelp home page GithubHelp logo

dod-advana / gamechanger-crawlers Goto Github PK

View Code? Open in Web Editor NEW
15.0 14.0 9.0 208.63 MB

GAMECHANGER Policy Analytics Site Crawlers

License: MIT License

Python 94.16% Dockerfile 1.39% Shell 4.45%
web-crawlers policy-as-code dod

gamechanger-crawlers's People

Contributors

ademouy avatar amaruca141 avatar antsega avatar ashermuse avatar brandonherzog avatar dakotahavel avatar domcritchlow avatar emmarez avatar kroop-olivia-bah avatar matthew-kersting avatar melkiga avatar mstyslinger avatar takao8 avatar vat99 avatar vctrstrm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gamechanger-crawlers's Issues

Exception when running dod issuances crawler

TypeError missing args. What to use for these args, or maybe I miss something?

Will parsed data be made publicly available? Save effort for contributors?

/Users/ilya/.virtualenvs/torch/lib/python3.8/site-packages/urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.esd.whs.mil'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn(
Traceback (most recent call last):
  File "/Users/ilya/pycharm_projects/rabin/gamechanger-crawlers/dataPipelines/gc_crawler/dod_issuances/cli.py", line 51, in <module>
    run()
  File "/Users/ilya/.virtualenvs/torch/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/ilya/.virtualenvs/torch/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/ilya/.virtualenvs/torch/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/ilya/.virtualenvs/torch/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/ilya/pycharm_projects/rabin/gamechanger-crawlers/dataPipelines/gc_crawler/dod_issuances/cli.py", line 46, in run
    for json_doc in results:
  File "/Users/ilya/pycharm_projects/rabin/gamechanger-crawlers/dataPipelines/gc_crawler/exec_model.py", line 89, in iter_validated_output_json
    for json_doc in self.iter_output_json():
  File "/Users/ilya/pycharm_projects/rabin/gamechanger-crawlers/dataPipelines/gc_crawler/exec_model.py", line 84, in iter_output_json
    for doc in self.iter_output_docs():
  File "/Users/ilya/pycharm_projects/rabin/gamechanger-crawlers/dataPipelines/gc_crawler/exec_model.py", line 79, in iter_output_docs
    for doc in self._parser.parse_docs_from_page(page_link, page_text):
  File "/Users/ilya/pycharm_projects/rabin/gamechanger-crawlers/dataPipelines/gc_crawler/dod_issuances/models.py", line 145, in parse_docs_from_page
    doc = Document(
TypeError: __init__() missing 3 required positional arguments: 'display_doc_type', 'display_org', and 'display_source'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.