dreamproit / bill-similarity Goto Github PK

This issue is to estimate the task to:

1. Process a bill with a similarity algorithm to return a list of similar bills in the form of a BillToBill model (e.g. on the investigate_simhashes branch: https://github.com/dreamproit/bill-similarity/pull/4/files)
2. Save the similar bills to the BillToBill table of billtitles-py (https://github.com/dreamproit/billtitles-py/blob/main/billtitles/models.py#L72). This uses the helper function, create_billtobill to save the billtobill data:
   https://github.com/dreamproit/billtitles-py/blob/main/billtitles/crud.py#L123
3. Create a pipeline to do this:
   a. Once for all bills
   b. Each time bills are updated by the uscongress bill scraper

Related to dreamproit/BillMap#13

This is the equivalent of the pipeline that is already working to populate the database for BillMap, described here:
https://github.com/dreamproit/bill-similarity/blob/investigate_simhashes/docs/SQL_APPROACH.adoc#current-data-pipeline-and-storage

In repository provided on start there were several files in xml format such as samples/congress/116/uslm/BILLS-116hconres9enr.xml etc.
And there were a pasing script to get sections from each bill.
But looks like that this script doesn't work with the other bills from the set we download via congress tool.
So the main question is:
How (where) can I get more, preferrably the whole set of bills that i can split to sections for further work?
May be (that's just my suggestion) we should transform the parsing script so it would parse that set?
Or there is some step of transformation that i still haven't found yet, isn't it?

Anyway the main point is to get more bills to get get more sections from them.

@dmytro-ustynov pls pay attention to this section when you file an issue:

While you are creating an issue you can link it to the GitHub Project.

And Then

When issue's already created you can add it to the sprint, estimate it and change its status.

This is a reminder for me (TODO)
Just a start issue for project onboarding.
some issues to explain and investigate:

• project structure, environment requirements
• parsing tools
• bill structure and what part it consist of

Related issues that are also helpfull to figure out in context of searching similarities.

• Simhashing and minhashing

The current simhash README says how to set up the system (good); and run all bills (good). What is missing is to know how to

• run a single bill (by bill name or by bill path?),
• run a batch of bills
• figure out what is the form of the output of the functions. Is it a list, a dictionary, a JSON object? What are the fields that are output and how do we associate the results with each bill?
• use the output to populate the billsim database

Also, we should combine this information with details of how the simhash + SQL query (by @alexbojko) works and what the differences are from pure simhash.