urigo / israeli-bank-scrapers-modern-schemas

A new architecture for Poalim bank scraper to hopefully merge into Israeli-bank-scrapers when they are ready

israeli-bank-scrapers-modern-schemas

A new architecture for Israeli bank scrapers to hopefully merge into Israeli-bank-scrapers when they are ready

This project is very heavily ispired by israeli-bank-scrapers. I was using their scrapers but needed to get:

1. The original and full responses from the scrapers and not a normalized one
2. Fully type safe, including the HTTP responses themselves
3. Varivied against schemas - I wanted to be notified if a bank started sending something a bit different and to understand exactly each field they are sending

I believe this project could be integrated into the original bank scrapers. Or as a dependency (all the additions above are optional with a config) or directly integrating the code.

Currently this library supports Bank Hapoalim, Bank Hapoalim business (with SMS login) and Isracard.

Should be easy to migrate more scrapers from israeli-bank-scrapers into this new form.

israeli-bank-scrapers-modern-schemas's Issues

Remove old node flags that are not relevant

like
--experimental-specifier-resolution=node
--experimental-json-modules

Should we take this code change?

Describe the bug

eshaham/israeli-bank-scrapers#738

Your Example Website or App

sadfsdf

Steps to Reproduce the Bug or Issue

Fetch more in a single request if possible

Expected behavior

Fetch more in a single request if possible?

Screenshots or Videos

No response

Platform

Not needed

Additional context

No response

סכמה

https://github.com/Urigo/new-bank-poalim-scraper/blob/bfbb752ab7e282308a3bbe1846f3cd054f0befe8/src/schemas/ILSCheckingTransactionsDataSchema.json#L9

הסכמה בנויה כרגע ככה:
{type: [everything else]}
ככה מתקבל אצלך המידע?

כנ"ל ForeignTransactionsSchema

multiScrapers new branch issues

on this commit,
make sure if accountAgreementOpeningDate field is required at accountDataSchema.json
(it is for me, but @Urigo removed it)

Use REST directly instead of scraping?

In all modern mobile apps for banking, they must use REST to fetch the data. In FIBI for example, it super simple because their website is shitty, but in mobile app they are using REST. Same with Pepper.

You can either use Charles for iOS (https://www.charlesproxy.com/documentation/ios/), or use Charles for Mac (with it's Custom SSL certificate, and install the profile on your mobile phone with a proxy. This way all requests are being decrypted and you can track the complete request on your Macbook, and log the requests and responses).

I did some experiments on the iOS app of Poalim, found the following for example (for getting information about the current account's transactions):

I think this might simplify the implementation, no? We can even do JSON schema and GraphQL Mesh for that...

Replace Nodemon and concurrently

Replace with tsup or Bob or something more modern

Replace Moment with a smaller and more modern dates library

Try to load only the necessary assets with Puppeteer to load faster

  graph TD;
      A-->B;
      A-->C;
      B-->D;
      C-->D;
      click C "https://www.github.com" _blank

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency @types/node to v20.11.30
• Update dependency inquirer to v9.2.16
• Update Node.js to v21.7.1
• Update dependency nodemon to v3.1.0
• Update dependency puppeteer to v22.6.0
• Update dependency typescript to v5.4.3
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

nodenv

.node-version

• node 21.6.2

npm

package.json

• ajv 8.12.0
• ajv-formats 2.1.1
• inquirer 9.2.15
• node-fetch 3.3.2
• puppeteer 22.1.0
• uuid 9.0.1
• @types/inquirer 9.0.7
• @types/node 20.11.19
• @types/node-fetch 3.0.3
• @types/uuid 9.0.8
• concurrently 8.2.2
• husky 9.0.11
• json-schema-to-typescript 13.1.2
• lint-staged 15.2.2
• nodemon 3.0.3
• prettier 3.2.5
• typescript 5.3.3
• node 21.6.2

---

• Check this box to trigger a request for Renovate to run again on this repository

Interface generating error

https://github.com/Urigo/new-bank-poalim-scraper/blob/bfbb752ab7e282308a3bbe1846f3cd054f0befe8/src/schemas/ILSCheckingTransactionsDataSchema.json#L42

wherever a schema property have only "const" and no "type", the generator creates [k: string]: unknown; definition

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.