GithubHelp home page GithubHelp logo

dap's Issues

Error binding parameter :defendant_email 

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
pipl match found
Traceback (most recent call last):
  File "identify_cases.py", line 45, in <module>
    main()
  File "identify_cases.py", line 41, in main
    identify()
  File "identify_cases.py", line 30, in identify
    defendant["zip"], defendant["email"], defendant["facebook"])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/database.py", line 143, in db_move_to_matched_cases
    'defendant_facebook': defendant_facebook})
sqlite3.InterfaceError: Error binding parameter :defendant_email - probably unsupported type.

Implement two-step PiPl e-mail collection

PiPl API keys have tiers, see https://pipl.com/api/pricing.

We are using contact tier keys for mailing address.

To get e-mail addresses we need business tier keys.

Instead of running all searches at business tier, we should implement a two step process.

  1. Contact tier gets most of our data.
  2. If contact tier returns the existence of an e-mail address (it does so by returning "[email protected]" as an e-mail), run another PiPl with the same search pointer at the business tier to get the e-mail.
  3. Push that e-mail address to the clicksend API.

https://github.com/sirredbeard/DAP/blob/master/api_interfaces.py#L157

We will deprecate use of social tier until Facebook messaging is sorted out.

database.py: name db_move_to_incomplete_pipl is not defined

GENERAL: pipl match found for: MCMAHON STACEY │ Traceback (most recent call last): │ File "main.py", line 10, in <module> │ import send_letters │ File "/home/grufwub/Git/other/DAP/send_letters.py", line 99, in <module> │ main() │ File "/home/grufwub/Git/other/DAP/send_letters.py", line 95, in main │ process_matches() │ File "/home/grufwub/Git/other/DAP/send_letters.py", line 89, in process_matches │ db_move_to_incomplete_pipl(case_number) │ NameError: name 'db_move_to_incomplete_pipl' is not defined

fix by adding this to the imports in send_letters.py -- will PR soon

Investigate pipl error and remediation 

PIPL: fetching request...
PIPL: No matching person found for FERGUS NICHOLAUS.
PIPL: FERGUS NICHOLAUS: {'match_true': False}
GENERAL: {'match_true': False}
GENERAL: no pipl match for: FERGUS NICHOLAUS
PIPL: {'person': <piplapis.data.containers.Person object at 0x7f8047195c88>, 'api_key': 'CONTACT-PREMIUM-83u2ema5vwg7302rrwvbarve', 'show_sources': None, 'live_feeds': None, 'minimum_match': None, 'top_match': None, 'minimum_probability': 0.8, 'hide_sponsored': None, 'match_requirements': None, 'source_category_requirements': None, 'use_https': True, 'infer_persons': None, 'response_class': <class 'piplapis.search.SearchAPIResponse'>}
PIPL: fetching request...
PIPL: direct match!
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    import identify_cases
  File "/home/dapadmin/DAP/identify_cases.py", line 52, in <module>
    main()
  File "/home/dapadmin/DAP/identify_cases.py", line 48, in main
    identify()
  File "/home/dapadmin/DAP/identify_cases.py", line 27, in identify
    defendant = api_pipl(possible_case[1])
  File "/home/dapadmin/DAP/api_interfaces.py", line 172, in api_pipl
    last_seen = address.last_seen
UnboundLocalError: local variable 'address' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
    dap_log_general(LogLevel.CRITICAL, "exception encountered [%i]: %s" % (e.errno, e.strerror))
NameError: name 'LogLevel' is not defined

AttributeError: 'Address' object has no attribute 'apt'

This is something I fucked up adding house and apartment numbers.

python3 identify_cases.py
Traceback (most recent call last):
  File "identify_cases.py", line 48, in <module>
    main()
  File "identify_cases.py", line 44, in main
    identify()
  File "identify_cases.py", line 25, in identify
    defendant = api_pipl(possible_case[1])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/api_interfaces.py", line 210, in api_pipl
    defendant["apt"] = defendant_address.apt if defendant_address.apt else ""
AttributeError: 'Address' object has no attribute 'apt'

Link to offending code: https://github.com/sirredbeard/DAP/blob/master/api_interfaces.py#L210

Investigate error and possible remediation 

GENERAL: resetting error counter
MAINFRAME: selecting CATS function
MAINFRAME: opening docket search page
MAINFRAME: searching mainframe for case: MC6938
MAINFRAME: checking to see if case exists
MAINFRAME: case found
MAINFRAME: parsing first page of case file
MAINFRAME: navigating additional pages of case file
MAINFRAME: getting party names
MAINFRAME: defendant counsel empty, setting 'NONE'
MAINFRAME: judge name: STEVEN D. SMITH
date filed: 08/15/2019
time filed:
civil action code: CV-ST
action description: CIVIL SUIT
defendant name: TALLMAN ASHLEY A
plaintiff name: PORTFOLIO REC/COM BK/VICTORIA'S S
plaintiff counsel: COOLING & WINTER, LLC
defendant counsel: NONE
GENERAL: checking data
GENERAL: writing case to NEW_CASE
GENERAL: resetting error counter
MAINFRAME: selecting CATS function
MAINFRAME: opening docket search page
MAINFRAME: searching mainframe for case: MC6939
MAINFRAME: checking to see if case exists
MAINFRAME: case found
MAINFRAME: parsing first page of case file
MAINFRAME: navigating additional pages of case file
MAINFRAME: getting party names
Traceback (most recent call last):
  File "main.py", line 7, in <module>
    import get_cases
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 110, in <module>
    main()
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 107, in main
    get_municipal_court()
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 97, in get_municipal_court
    each_court(court_name)
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 66, in each_court
    last_successful_case_number)  # scan, giving back last known successfully accessed case number
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 36, in scan
    year, judge_name, date_filed, time_filed, plaintiff_name, plaintiff_counsel, defendant_name, defendant_counsel, civil_action, action_description = mainframe_parse_case()  # pull the data from mainframe
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/mainframe.py", line 110, in mainframe_parse_case
    name = em.string_get(x, 38, 33).strip()
  File "/home/hayden/.local/lib/python3.7/site-packages/py3270/__init__.py", line 455, in string_get
    return cmd.data[0].decode("ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 32: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
    dap_log_general(LogLevel.CRITICAL, "exception encountered [%i]: %s" % (e.errno, e.strerror))
NameError: name 'LogLevel' is not defined

TypeError: %i format: a number is required, not datetime.datetime

This is to check out the:

dap_log(log_type=LogType.PIPL, log_level=LogLevel.WARN, message=message)
File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/dap_logging.py", line 121, in dap_log
message = "%i [%s] %s: %s" % (datetime.now(), session_log_no, log_type.name, message)
TypeError: %i format: a number is required, not datetime.datetime

Part.

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
Traceback (most recent call last):
  File "identify_cases.py", line 48, in <module>
    main()
  File "identify_cases.py", line 44, in main
    identify()
  File "identify_cases.py", line 25, in identify
    defendant = api_pipl(possible_case[1])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/api_interfaces.py", line 103, in api_pipl
    dap_log(log_type=LogType.PIPL, log_level=LogLevel.WARN, message=message)
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/dap_logging.py", line 121, in dap_log
    message = "%i [%s] %s: %s" % (datetime.now(), session_log_no, log_type.name, message)
TypeError: %i format: a number is required, not datetime.datetime

Handle certain classes of names

So this is actually funny and interesting.

When a car has been abandoned at a towing yard for so long that the storage fees exceed the value of the car the impound company is allowed to take and resell the car.

Mechanics are allowed to do this too actually, if a car they did work on has been abandoned and they haven't been paid, though they do it less often.

When you do this they have to get a local judge's approval they did their best to find the owner, the storage fees are valid, etc. and they do this by filing a lawsuit.

They have to give the lawsuit a case caption, so-and-so v. someone, but in these cases there really isn't a someone, the car has been abandoned, but they need to give people notice that this case is happening involving this car, in case there is a legitimate owner somewhere.

So they name the cases impound lot vs. 2008 TOYOTA CAMRY 4T1BE46K18U2525. No joke.

We should detect this somehow and prevent them from getting into possible cases because PiPl obviously can't find them.

I think Python has some pretty powerful pattern matching, so I think we could do a check for a NNNN [word] [word] [partial VIN].

GENERAL: pipl match found for: HOLLAND MARGARITA
PIPL: invalid name format: 2012 HONDA CIVIC 19XFB2F5XCE04457
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2012 HONDA CIVIC 19XFB2F5XCE04457
PIPL: invalid name format: 2007 CHRYSLER SEBRING 1C3LC56K77N
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2007 CHRYSLER SEBRING 1C3LC56K77N
PIPL: invalid name format: 2007 DODGE RAM 1500 1D7HA18P47S10
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2007 DODGE RAM 1500 1D7HA18P47S10
PIPL: invalid name format: 2009 TOYOTA CAMRY 4T4BE46K59R1270
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2009 TOYOTA CAMRY 4T4BE46K59R1270
PIPL: invalid name format: 2004 NISSAN ALTIMA 1N4AL11D14C126
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2004 NISSAN ALTIMA 1N4AL11D14C126
PIPL: invalid name format: 2017 KIA RIO KNADM4A37H6037185
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2017 KIA RIO KNADM4A37H6037185
PIPL: invalid name format: 2003 BMW 325I WBAEV33443KL79999
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2003 BMW 325I WBAEV33443KL79999
PIPL: invalid name format: 2012 NISSAN SENTRA 3N1AB6AP7CL643
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2012 NISSAN SENTRA 3N1AB6AP7CL643
PIPL: invalid name format: 2004 FORD EXPLORER 1FMZU63K14UA08
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2004 FORD EXPLORER 1FMZU63K14UA08
PIPL: invalid name format: 2004 JEEP GRND CHEROKE 1J4GX48S44
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2004 JEEP GRND CHEROKE 1J4GX48S44
PIPL: invalid name format: 2008 TOYOTA CAMRY 4T1BE46K18U2525
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2008 TOYOTA CAMRY 4T1BE46K18U2525
PIPL: invalid name format: 2011 MITSUBISHI GALANT 4A32B2FF9B
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2011 MITSUBISHI GALANT 4A32B2FF9B

NameError: name 'session_log_no' is not defined, dap_logging.py", line 121, in dap_log 

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
Traceback (most recent call last):
  File "identify_cases.py", line 45, in <module>
    main()
  File "identify_cases.py", line 41, in main
    identify()
  File "identify_cases.py", line 25, in identify
    defendant = api_pipl(possible_case[1])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/api_interfaces.py", line 214, in api_pipl
    dap_log(log_type=LogType.PIPL, log_level=LogLevel.INFO, message=str(defendant))
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/dap_logging.py", line 121, in dap_log
    message = "%i [%s] %s: %s" % (datetime.now(), session_log_no, log_type.name, message)
NameError: name 'session_log_no' is not defined

More graceful failures

Below something went wrong and a defendant name wasn't read.

When something isn't read from the mainframe screen, it means there's been a glitch in the mainframe.

There is already detection for when this occurs at this step, see here https://github.com/sirredbeard/DAP/blob/master/mainframe.py#L133

We need to add code there to define any blank variables as "ERROR" to avoid the error below.

Then add detection this those "ERROR" and a graceful shutdown, abandoning that case, and making sure the correct last_successful_case_number is still passed here https://github.com/sirredbeard/DAP/blob/master/get_cases.py#L35.

We should also add an extra random wait next to this one to reduce this error occurring in the first place https://github.com/sirredbeard/DAP/blob/master/mainframe.py#L105

civil action code CV-DP
action description DISPOSSESS
getting the parties names
defendant name HOUSTON KENDELL
plaintiff name RIVERSTONE COMMERCIAL PROPERTIES.
plaintiff counsel NONE
defendant counsel NONE
writing case to NEW_CASE
resetting error counter
selecting CATS function
opening docket search page
searching mainframe for case:MC6882
checking to see if case exists
case found
parsing first page of case file
judge name STEVEN D. SMITH
date filed 08/13/2019
time filed
navigating additional pages of case file
civil action code
action description
getting the parties names
defendant name empty
Traceback (most recent call last):
  File "get_cases.py", line 101, in <module>
    main()
  File "get_cases.py", line 98, in main
    get_municipal_court()
  File "get_cases.py", line 88, in get_municipal_court
    each_court(court_name)
  File "get_cases.py", line 57, in each_court
    last_successful_case_number)  # scan, giving back last known successfully accessed case number
  File "get_cases.py", line 34, in scan
    year, judge_name, date_filed, time_filed, plaintiff_name, plaintiff_counsel, defendant_name, defendant_counsel, civil_action, action_description = mainframe_parse_case()  # pull the data from mainframe
TypeError: cannot unpack non-iterable NoneType object

get_cases.py not checking for 35 cases after retrieving a case successfully

  1. Searches for SU 2077, doesn't find, moves on
  2. Searches for SU 2078, finds, processes, and should move onto try SU 2079
  3. Should try SU 2079 through SU 2114
  4. Instead tries SC 863
hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 get_cases.py
getting superior court cases
getting latest case number
connecting to mainframe
pausing for 1 seconds
selecting mainframe application
pausing for 1 seconds
logging into mainframe
sending username
pausing for 1 seconds
sending password
pausing for 1 seconds
checking if user is already logged in
login successful
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SU2077
pausing for 1 seconds
checking to see if case exists
case not found
pausing for 1 seconds
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SU2078
pausing for 1 seconds
checking to see if case exists
case found
parsing first page of case file
judge name RON MULLINS
date filed 08/09/2019
time filed 11:36
navigating additional pages of case file
pausing for 1 seconds
pausing for 1 seconds
getting the parties names
defendant name JACKSON JACKSON
plaintiff name DISCOVER BANK
pausing for 1 seconds
recording last successful case number: SU-2019-CV-2078
getting state court cases
getting latest case number
connecting to mainframe
pausing for 1 seconds
selecting mainframe application
pausing for 1 seconds
logging into mainframe
sending username
pausing for 1 seconds
sending password
pausing for 1 seconds
checking if user is already logged in
login successful
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SC863
pausing for 1 seconds
checking to see if case exists
case not found
pausing for 1 seconds
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SC864
pausing for 1 seconds
checking to see if case exists
case not found

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.