sirredbeard / dap

Debtor Advice Program

DAP

Debtor Advice Program

Scans a court IBM mainframe for recent cases filed by creditors and sends debtors a letter or e-mail about their legal rights.

• Because the mainframe does not provide an address for the defendant, a search is run using the pipl API
• The letter is sent by the lob API
• The e-mail is sent by the clicksend API, which unlike plain smtp reduces likelihood of being marked spam

Dependencies

$ sudo apt-get install git python3 python3-pip s3270 x3270 -y

Note: s3270 is in non-free repository. x3270 is useful for debugging mainframe interactions but not required, see comments in mainframe.py.

$ pip3 install py3270 lob piplapis-python
$ pip3 install git+https://github.com/ClickSend/clicksend-python.git

Components

main.py

- calls get_cases.py
- calls screen_cases.py
- calls identify_cases.py
- calls send_letters.py

```
NOTE: These four scripts are meant to be run consecutavely in the order above.
Each step is designed that a failure of one script will not result in data loss.
Any unprocessed data will simply be picked up and handled in the next pass.
This script is meant to run every 2-3 hours.
```

get_cases.py

- checks for new cases in all 4 local courts by connecting to mainframe, using py3720 and an internal api in mainframe.py
- keeps a database of existing case numbers in CASENUMBERS, because case numbers are issued sequentially by year, searches start with last known good+1 and fail after 15 cases are not found
- stores data from new cases for analysis in NEW_CASE in dap.sqlite

screen_cases.py

- opens new cases in NEW_CASE
- screens plaintiff_name for keywords and known creditors which are listed as CREDITOR
- if a match is found, case is moved to possible cases in POSSIBLE_CASE
- if a match is not found, case is moved to rejected cases in REJECTEDCASES and noted why

identify_cases.py

- opens possible cases in POSSIBLE_CASE
- matches cases in POSSIBLE_CASE to people using the pipl api, via api_interfaces.py
- if a match is found, case is moved to possible cases in POSSIBLE_CASE
- if a match is not found, case is moved to rejected cases in REJECTEDCASES and noted why

send_letters.py

- processes MATCHEDCASES
- send prospective debtors a letter, e-mail, and/or facebook message based on available data
- sends letter via USPS via lob api
- sends e-mail via clicksend api
- saves processed cases to PROCESSEDCASES with time-stamp for any/all of above

api_interfaces.py

- contains functions for interacting with various apis

api_keys.py (only .example uploaded to github)

- contains api keys for apis

database.py

- contains functions for interacting with local sqlite database

mainframe.py

- contains functions for interacting with court mainframe

mainframe_credentials.py (only .example uploaded to github)

- stores contains mainframe credentials

dap_logging.py

- contains functions for logging application events

- check file for usage information

combine_logs.py

- combine all logs in logging directory and order based on session log count

dap.sqlite, tables:

- CASENUMBERS - for tracking last known case numbers
- NEW_CASE - all cases scanned from mainframe
- POSSIBLE_CASE - cases whose plaintiff contains a keyword from CREDITOR
- MATCHED_CASE - cases whose defendant has been matched to a person
- PROCESSED_CASE - cases that were sent letter, e-mail, or facebook message, with time-stamp
- REJECTED_CASE - cases rejected for no matching creditor or individual
- CREDITOR - list of creditors and keywords for creditors

database.sql:

- creates dap.sqlite database from scratch

test_data.sql:

- populates dap.sqlite database with test data

logs:

- mainframe.log - for logging mainframe responses
- database.log - for logging database events
- pipl.log - for logging pipl api responses
- clicksend.log - for logging clicksend api responses
- lob.log - for logging lob api responses
- compliance.log - for logging compliance-related activities

dap's Issues

Implement two-step PiPl e-mail collection

PiPl API keys have tiers, see https://pipl.com/api/pricing.

We are using contact tier keys for mailing address.

To get e-mail addresses we need business tier keys.

Instead of running all searches at business tier, we should implement a two step process.

1. Contact tier gets most of our data.
2. If contact tier returns the existence of an e-mail address (it does so by returning "[email protected]" as an e-mail), run another PiPl with the same search pointer at the business tier to get the e-mail.
3. Push that e-mail address to the clicksend API.

https://github.com/sirredbeard/DAP/blob/master/api_interfaces.py#L157

We will deprecate use of social tier until Facebook messaging is sorted out.

NameError: name 'session_log_no' is not defined, dap_logging.py", line 121, in dap_log

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
Traceback (most recent call last):
  File "identify_cases.py", line 45, in <module>
    main()
  File "identify_cases.py", line 41, in main
    identify()
  File "identify_cases.py", line 25, in identify
    defendant = api_pipl(possible_case[1])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/api_interfaces.py", line 214, in api_pipl
    dap_log(log_type=LogType.PIPL, log_level=LogLevel.INFO, message=str(defendant))
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/dap_logging.py", line 121, in dap_log
    message = "%i [%s] %s: %s" % (datetime.now(), session_log_no, log_type.name, message)
NameError: name 'session_log_no' is not defined

Investigate pipl error and remediation

PIPL: fetching request...
PIPL: No matching person found for FERGUS NICHOLAUS.
PIPL: FERGUS NICHOLAUS: {'match_true': False}
GENERAL: {'match_true': False}
GENERAL: no pipl match for: FERGUS NICHOLAUS
PIPL: {'person': <piplapis.data.containers.Person object at 0x7f8047195c88>, 'api_key': 'CONTACT-PREMIUM-83u2ema5vwg7302rrwvbarve', 'show_sources': None, 'live_feeds': None, 'minimum_match': None, 'top_match': None, 'minimum_probability': 0.8, 'hide_sponsored': None, 'match_requirements': None, 'source_category_requirements': None, 'use_https': True, 'infer_persons': None, 'response_class': <class 'piplapis.search.SearchAPIResponse'>}
PIPL: fetching request...
PIPL: direct match!
Traceback (most recent call last):
  File "main.py", line 9, in <module>
    import identify_cases
  File "/home/dapadmin/DAP/identify_cases.py", line 52, in <module>
    main()
  File "/home/dapadmin/DAP/identify_cases.py", line 48, in main
    identify()
  File "/home/dapadmin/DAP/identify_cases.py", line 27, in identify
    defendant = api_pipl(possible_case[1])
  File "/home/dapadmin/DAP/api_interfaces.py", line 172, in api_pipl
    last_seen = address.last_seen
UnboundLocalError: local variable 'address' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
    dap_log_general(LogLevel.CRITICAL, "exception encountered [%i]: %s" % (e.errno, e.strerror))
NameError: name 'LogLevel' is not defined

Finish logging implementation

Remove leading 0 from address printed in Date Filed

When a letter is sent, need to move from MATCHED_CASES to PROCESSED_CASES

https://github.com/sirredbeard/DAP/blob/master/send_letters.py#L88

Handle certain classes of names

So this is actually funny and interesting.

When a car has been abandoned at a towing yard for so long that the storage fees exceed the value of the car the impound company is allowed to take and resell the car.

Mechanics are allowed to do this too actually, if a car they did work on has been abandoned and they haven't been paid, though they do it less often.

When you do this they have to get a local judge's approval they did their best to find the owner, the storage fees are valid, etc. and they do this by filing a lawsuit.

They have to give the lawsuit a case caption, so-and-so v. someone, but in these cases there really isn't a someone, the car has been abandoned, but they need to give people notice that this case is happening involving this car, in case there is a legitimate owner somewhere.

So they name the cases impound lot vs. 2008 TOYOTA CAMRY 4T1BE46K18U2525. No joke.

We should detect this somehow and prevent them from getting into possible cases because PiPl obviously can't find them.

I think Python has some pretty powerful pattern matching, so I think we could do a check for a NNNN [word] [word] [partial VIN].

GENERAL: pipl match found for: HOLLAND MARGARITA
PIPL: invalid name format: 2012 HONDA CIVIC 19XFB2F5XCE04457
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2012 HONDA CIVIC 19XFB2F5XCE04457
PIPL: invalid name format: 2007 CHRYSLER SEBRING 1C3LC56K77N
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2007 CHRYSLER SEBRING 1C3LC56K77N
PIPL: invalid name format: 2007 DODGE RAM 1500 1D7HA18P47S10
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2007 DODGE RAM 1500 1D7HA18P47S10
PIPL: invalid name format: 2009 TOYOTA CAMRY 4T4BE46K59R1270
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2009 TOYOTA CAMRY 4T4BE46K59R1270
PIPL: invalid name format: 2004 NISSAN ALTIMA 1N4AL11D14C126
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2004 NISSAN ALTIMA 1N4AL11D14C126
PIPL: invalid name format: 2017 KIA RIO KNADM4A37H6037185
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2017 KIA RIO KNADM4A37H6037185
PIPL: invalid name format: 2003 BMW 325I WBAEV33443KL79999
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2003 BMW 325I WBAEV33443KL79999
PIPL: invalid name format: 2012 NISSAN SENTRA 3N1AB6AP7CL643
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2012 NISSAN SENTRA 3N1AB6AP7CL643
PIPL: invalid name format: 2004 FORD EXPLORER 1FMZU63K14UA08
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2004 FORD EXPLORER 1FMZU63K14UA08
PIPL: invalid name format: 2004 JEEP GRND CHEROKE 1J4GX48S44
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2004 JEEP GRND CHEROKE 1J4GX48S44
PIPL: invalid name format: 2008 TOYOTA CAMRY 4T1BE46K18U2525
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2008 TOYOTA CAMRY 4T1BE46K18U2525
PIPL: invalid name format: 2011 MITSUBISHI GALANT 4A32B2FF9B
GENERAL: {'match_true': False}
GENERAL: no pipl match for: 2011 MITSUBISHI GALANT 4A32B2FF9B

More graceful failures

Below something went wrong and a defendant name wasn't read.

When something isn't read from the mainframe screen, it means there's been a glitch in the mainframe.

There is already detection for when this occurs at this step, see here https://github.com/sirredbeard/DAP/blob/master/mainframe.py#L133

We need to add code there to define any blank variables as "ERROR" to avoid the error below.

Then add detection this those "ERROR" and a graceful shutdown, abandoning that case, and making sure the correct last_successful_case_number is still passed here https://github.com/sirredbeard/DAP/blob/master/get_cases.py#L35.

We should also add an extra random wait next to this one to reduce this error occurring in the first place https://github.com/sirredbeard/DAP/blob/master/mainframe.py#L105

civil action code CV-DP
action description DISPOSSESS
getting the parties names
defendant name HOUSTON KENDELL
plaintiff name RIVERSTONE COMMERCIAL PROPERTIES.
plaintiff counsel NONE
defendant counsel NONE
writing case to NEW_CASE
resetting error counter
selecting CATS function
opening docket search page
searching mainframe for case:MC6882
checking to see if case exists
case found
parsing first page of case file
judge name STEVEN D. SMITH
date filed 08/13/2019
time filed
navigating additional pages of case file
civil action code
action description
getting the parties names
defendant name empty
Traceback (most recent call last):
  File "get_cases.py", line 101, in <module>
    main()
  File "get_cases.py", line 98, in main
    get_municipal_court()
  File "get_cases.py", line 88, in get_municipal_court
    each_court(court_name)
  File "get_cases.py", line 57, in each_court
    last_successful_case_number)  # scan, giving back last known successfully accessed case number
  File "get_cases.py", line 34, in scan
    year, judge_name, date_filed, time_filed, plaintiff_name, plaintiff_counsel, defendant_name, defendant_counsel, civil_action, action_description = mainframe_parse_case()  # pull the data from mainframe
TypeError: cannot unpack non-iterable NoneType object

api_interfaces: implement picking most likely candidate from pipl possible matches

at the moment only picks first from the list of possible matches as placeholder code

Implement sending e-mail (after #34 is implemented)

get_cases.py not checking for 35 cases after retrieving a case successfully

1. Searches for SU 2077, doesn't find, moves on
2. Searches for SU 2078, finds, processes, and should move onto try SU 2079
3. Should try SU 2079 through SU 2114
4. Instead tries SC 863

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 get_cases.py
getting superior court cases
getting latest case number
connecting to mainframe
pausing for 1 seconds
selecting mainframe application
pausing for 1 seconds
logging into mainframe
sending username
pausing for 1 seconds
sending password
pausing for 1 seconds
checking if user is already logged in
login successful
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SU2077
pausing for 1 seconds
checking to see if case exists
case not found
pausing for 1 seconds
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SU2078
pausing for 1 seconds
checking to see if case exists
case found
parsing first page of case file
judge name RON MULLINS
date filed 08/09/2019
time filed 11:36
navigating additional pages of case file
pausing for 1 seconds
pausing for 1 seconds
getting the parties names
defendant name JACKSON JACKSON
plaintiff name DISCOVER BANK
pausing for 1 seconds
recording last successful case number: SU-2019-CV-2078
getting state court cases
getting latest case number
connecting to mainframe
pausing for 1 seconds
selecting mainframe application
pausing for 1 seconds
logging into mainframe
sending username
pausing for 1 seconds
sending password
pausing for 1 seconds
checking if user is already logged in
login successful
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SC863
pausing for 1 seconds
checking to see if case exists
case not found
pausing for 1 seconds
selecting CATS function
pausing for 1 seconds
opening docket search page
pausing for 1 seconds
searching mainframe for case:SC864
pausing for 1 seconds
checking to see if case exists
case not found

Move matched cases without addresses to REJECTED_CASE

AttributeError: 'Address' object has no attribute 'apt'

This is something I fucked up adding house and apartment numbers.

python3 identify_cases.py
Traceback (most recent call last):
  File "identify_cases.py", line 48, in <module>
    main()
  File "identify_cases.py", line 44, in main
    identify()
  File "identify_cases.py", line 25, in identify
    defendant = api_pipl(possible_case[1])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/api_interfaces.py", line 210, in api_pipl
    defendant["apt"] = defendant_address.apt if defendant_address.apt else ""
AttributeError: 'Address' object has no attribute 'apt'

Link to offending code: https://github.com/sirredbeard/DAP/blob/master/api_interfaces.py#L210

When a Pipl match is not found, need to move case from POSSIBLE_CASES to REJECTED_CASES

https://github.com/sirredbeard/DAP/blob/master/identify_cases.py#L33

Investigate Facebook options

On first glance I didn't see an option to send from page, only user.

Although you may need to log in as the user and then select the page with a thread setting.

https://github.com/sirredbeard/DAP/blob/master/api_interfaces.py#L342

We can't use the official Messenger API for this because they are unsolicited.

https://developers.facebook.com/docs/messenger-platform/send-messages/#message_types

TypeError: %i format: a number is required, not datetime.datetime

This is to check out the:

dap_log(log_type=LogType.PIPL, log_level=LogLevel.WARN, message=message)
File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/dap_logging.py", line 121, in dap_log
message = "%i [%s] %s: %s" % (datetime.now(), session_log_no, log_type.name, message)
TypeError: %i format: a number is required, not datetime.datetime

Part.

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
Traceback (most recent call last):
  File "identify_cases.py", line 48, in <module>
    main()
  File "identify_cases.py", line 44, in main
    identify()
  File "identify_cases.py", line 25, in identify
    defendant = api_pipl(possible_case[1])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/api_interfaces.py", line 103, in api_pipl
    dap_log(log_type=LogType.PIPL, log_level=LogLevel.WARN, message=message)
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/dap_logging.py", line 121, in dap_log
    message = "%i [%s] %s: %s" % (datetime.now(), session_log_no, log_type.name, message)
TypeError: %i format: a number is required, not datetime.datetime

api_interfaces: sometimes pipl requests get stuck > 15minutes with no response

maybe there's some kind of exception or issue we're not catching?

database.py: name db_move_to_incomplete_pipl is not defined

GENERAL: pipl match found for: MCMAHON STACEY │ Traceback (most recent call last): │ File "main.py", line 10, in <module> │ import send_letters │ File "/home/grufwub/Git/other/DAP/send_letters.py", line 99, in <module> │ main() │ File "/home/grufwub/Git/other/DAP/send_letters.py", line 95, in main │ process_matches() │ File "/home/grufwub/Git/other/DAP/send_letters.py", line 89, in process_matches │ db_move_to_incomplete_pipl(case_number) │ NameError: name 'db_move_to_incomplete_pipl' is not defined

fix by adding this to the imports in send_letters.py -- will PR soon

Add note when there are no jobs to process

cronjob.log

Investigate error and possible remediation

GENERAL: resetting error counter
MAINFRAME: selecting CATS function
MAINFRAME: opening docket search page
MAINFRAME: searching mainframe for case: MC6938
MAINFRAME: checking to see if case exists
MAINFRAME: case found
MAINFRAME: parsing first page of case file
MAINFRAME: navigating additional pages of case file
MAINFRAME: getting party names
MAINFRAME: defendant counsel empty, setting 'NONE'
MAINFRAME: judge name: STEVEN D. SMITH
date filed: 08/15/2019
time filed:
civil action code: CV-ST
action description: CIVIL SUIT
defendant name: TALLMAN ASHLEY A
plaintiff name: PORTFOLIO REC/COM BK/VICTORIA'S S
plaintiff counsel: COOLING & WINTER, LLC
defendant counsel: NONE
GENERAL: checking data
GENERAL: writing case to NEW_CASE
GENERAL: resetting error counter
MAINFRAME: selecting CATS function
MAINFRAME: opening docket search page
MAINFRAME: searching mainframe for case: MC6939
MAINFRAME: checking to see if case exists
MAINFRAME: case found
MAINFRAME: parsing first page of case file
MAINFRAME: navigating additional pages of case file
MAINFRAME: getting party names
Traceback (most recent call last):
  File "main.py", line 7, in <module>
    import get_cases
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 110, in <module>
    main()
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 107, in main
    get_municipal_court()
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 97, in get_municipal_court
    each_court(court_name)
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 66, in each_court
    last_successful_case_number)  # scan, giving back last known successfully accessed case number
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/get_cases.py", line 36, in scan
    year, judge_name, date_filed, time_filed, plaintiff_name, plaintiff_counsel, defendant_name, defendant_counsel, civil_action, action_description = mainframe_parse_case()  # pull the data from mainframe
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/mainframe.py", line 110, in mainframe_parse_case
    name = em.string_get(x, 38, 33).strip()
  File "/home/hayden/.local/lib/python3.7/site-packages/py3270/__init__.py", line 455, in string_get
    return cmd.data[0].decode("ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 32: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 12, in <module>
    dap_log_general(LogLevel.CRITICAL, "exception encountered [%i]: %s" % (e.errno, e.strerror))
NameError: name 'LogLevel' is not defined

Error binding parameter :defendant_email

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
pipl match found
Traceback (most recent call last):
  File "identify_cases.py", line 45, in <module>
    main()
  File "identify_cases.py", line 41, in main
    identify()
  File "identify_cases.py", line 30, in identify
    defendant["zip"], defendant["email"], defendant["facebook"])
  File "/mnt/c/Users/Hayden/OneDrive/Documents/GitHub/DAP/database.py", line 143, in db_move_to_matched_cases
    'defendant_facebook': defendant_facebook})
sqlite3.InterfaceError: Error binding parameter :defendant_email - probably unsupported type.

Reminder: past issues (etc) may have personal information from tracebacks, screenshots etc

This is just a reminder issue to leave in place so we can go back and delete private information, should we open-source this in the future.

identify_cases.py moved SU 2078 from POSSIBLE_CASE to MATCHED_CASE without storing Pipl data (address, etc.)

hayden@T470s:~/winhome/OneDrive/Documents/GitHub/DAP$ python3 identify_cases.py
pipl match found

dap.sqlite.zip.zip