Support data source pagination about li

[camjc]

JP is a good example, we hit the 10,000 limit there

[camjc]

https://services8.arcgis.com/JdxivnCyd1rvJTrY/arcgis/rest/services/v2_covid19_list_csv/FeatureServer/0/query lets you query the dataset from a UI. Here are the main settings to get the JSON we use

[camjc]

Hoping someone can advise on how the pagination works, I don't know how it does.

[ryanblock]

Agreed. I know nothing about this system. I could really use:

1. A broken source that needs this
2. Clear instructions on how it's broken (eg steps to repro)
3. If available, any ideas on how to unbreak things and get the source humming

[jzohrab]

Perhaps relevant: covidatlas/coronadatascraper#839 - "WIP: add scraper for Singapore ARCGIS" There are a few PRs for arcgis pagination open in CDS, hard to sort out what's what though. El mié., 13 de may. de 2020 a la(s) 18:02, Ryan Block ( [email protected]) escribió:

…

Agreed. I know nothing about this system. I could really use: 1. A broken source that needs this 2. Clear instructions on how it's broken (eg steps to repro) 3. If available, any ideas on how to unbreak things and get the source humming — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMPWDOANEH2DYRIWWXO2C3RRMKHXANCNFSM4MKGX6LA> .

[jzohrab]

Actually relevant: currently, the PA scraper fetches paginated data. It handles this itself. e.g.

$ yarn start --location PA
...
1000 records from "... url ...&resultOffset=0&resultRecordCount=50000&f=json
...
✏️  coronadatascraper-cache/2020-5-17/55c884a3dc1f7fe60c5bb08af5371500.json written
1000 records from "... url ... &resultOffset=1000&resultRecordCount=50000&f=json
...
✏️  coronadatascraper-cache/2020-5-17/de03720e752a8e6478e066e3cb308ee2.json written
1000 records from "... url ... &resultOffset=2000&resultRecordCount=50000&f=json
...

ref src/shared/scrapers/PA/
method async function TEMPfetchArcGISJSON(obj, featureURL, date) {

[jzohrab]

WIP PR will soon close this: https://github.com/covidatlas/li/pull/193/files

[jzohrab]

New PR: #218.

[jzohrab]

PR merged 🎉