Don't bunde the data about covid19_scenarios

[r-s-rai]

Hello,

Would it be possible to schedule a meeting about how you want the data implemented? My group and I were thinking about using an Amazon S3 bucket to store the data, but we wanted to coordinate with you guys since you'll be the ones in charge of the S3 bucket and managing it.

[rneher]

sure, let's discuss. How about tomorrow (Thu) late afternoon CEST, morning East Coast?

[r-s-rai]

That sounds good. Do you want to hold the meeting over Zoom, and what exact time would you like to meet?

…

________________________________ From: Richard Neher <[email protected]> Sent: Wednesday, July 1, 2020 3:04 AM To: neherlab/covid19_scenarios <[email protected]> Cc: Rai, Rohan <[email protected]>; Comment <[email protected]> Subject: Re: [neherlab/covid19_scenarios] Don't bunde the data (#744) sure, let's discuss. How about tomorrow (Thu) late afternoon CEST, morning East Coast? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#744 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKI6K4GX5GH65Z7SF6JKYCLRZLUY5ANCNFSM4N64YCZA>.

[ivan-aksamentov]

@r-s-rai Hello, sorry for missing the call.

I've setup a S3 bucket + cloudfront distribution + domain.

So the data is ready to be fetched from:
https://data.covid19-scenarios.org/ageDistribution.json
https://data.covid19-scenarios.org/scenarios.json
https://data.covid19-scenarios.org/caseCounts.json
https://data.covid19-scenarios.org/severityDistributions.json

This is just the contents of the src/assets/data directory.
If you replace the corresponding imports in src/io/defaults/get* with fetches (e.g. using axios) this should do the trick.

CORS is enabled in both S3 and Cloudfront. However preflight (OPTIONS) requests will probably not work. So keep that in mind.

Note however that this simple solution would introduce all kinds of new issues:

• what if data schema needs to be changed? (like in #743) Developers cannot just modify the files on the bucket, because it will take down the production site.
• previously data was guaranteed to be always there, in the bundle. Now fetch can fail for any reason and this will require additional plumbing to mitigate.
• requests are now done in series: first the bundle, then the data. If we proceed with #743 the request chain will only become longer. Each request introduces additional latency.

So the entire adventure is probably more complicated than swapping the imports with requests.

Okay, sounds bad, but are there any other alternatives? I don't know.
So why don't you give it a try and we will see where it goes.
Please open a (draft) pull request early on to keep the discussion going.

Let me know if you have any questions or if you encounter any problems (especially with the AWS setup).

cc @rneher

[rneher]

@r-s-rai -- as you see in Ivan's comments above, the operation turns out to be slightly trickier than anticipated. We suggest starting first with just replacing bundling of the jsons by fetching. from there, one could then move towards fetching the case counts one-by-one.

[r-s-rai]

Hello Ivan,

Would it be possible to schedule a meeting with you sometime soon to discuss the code?

Thank you,

Rohan Rai

[DaveedaKinG]

Hello Rai

I think it’ll be a really nice idea to discuss
With you on this

thanks

DaveedaKinG