Comments (17)
@zachlatta - Wasn't able to figure out how exactly data is stored, but I built a simple endpoint for the Johns Hopkins data repository that follows the same schema this project currently does.
For countries with multiple regions, all the regions data is aggregated into one.
Just GET with the country's ISO code. Eg:
https://covid-data--jajoosam.repl.co/iso/in gives
{
"country": "India",
"lastUpdated": "Sat, 21 Mar 2020 09:34:39 GMT",
"data": [{
"date": "3/18/20",
"totalCases": 156,
"newCases": 14,
"totalDeaths": 3,
"newDeaths": 0,
"currentInfected": 150
},
{
"date": "3/19/20",
"totalCases": 194,
"newCases": 38,
"totalDeaths": 4,
"newDeaths": 1,
"currentInfected": 186
},
{
"date": "3/20/20",
"totalCases": 244,
"newCases": 50,
"totalDeaths": 5,
"newDeaths": 1,
"currentInfected": 234
}]
}
from predictcovid.
Absolutely. Are there particular areas you’re interested in contributing to?
The #1 need right now is to switch. The data source from scraping Worldometers to the Johns Hopkins data repository they’re now publishing on GitHub and updating daily.
That will allow us to support any country publishing numbers: not just the few that Worldometers supports on their website (which are the ones currently on the site).
Would you be interested in taking that on?
from predictcovid.
Hi, I would like to contribute to include India in the dashboard. We have a billion people here and we need prediction for people to start taking precaution now!
from predictcovid.
@williamluke4: High-level, logic makes sense, but can you please adapt it for the schema we currently have going? You can see it at https://github.com/lachlanjc/covid19/blob/master/api/prisma/schema.prisma.
If you can do that and are up for the task, a pull request would be greatly appreciated and I would be happy to prioritize and merge. This would also enable support for India, which would meet @kn-neeraj's need (which I think would be a fantastic addition).
from predictcovid.
@zachlatta trying to figure this out. Comfortable with Python more than javascript. But let me figure out how to help.
@williamluke4 good work on the above script. Are you working on adapting it to schema? Let me know if you need any help
from predictcovid.
I'm pretty swamped atm, so if someone else could take over that would be great. Great work @jajoosam
from predictcovid.
Do you have a link to the Github?
from predictcovid.
@williamluke4: It's https://github.com/CSSEGISandData/COVID-19. I'm thinking we should be scraping from https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports.
from predictcovid.
@zachlatta Ok perfect, ill have a look and let you know my thoughts :)
from predictcovid.
from predictcovid.
Thoughts
- Query
csse_covid_19_data/csse_covid_19_daily_reports
to get all the files
interface File {
name: string;
path: string;
sha: string;
size: string;
url: string;
html_url: string;
git_url: string;
download_url: string;
type: string;
_links: {
self: string;
git: string;
html: string;
}
}
async function fetchFiles(){
const response = await fetch("https://api.github.com/repos/CSSEGISandData/COVID-19/contents/csse_covid_19_data/csse_covid_19_daily_reports")
const files: File[] = await response.json();
const csvFiles = files.filter(file => file.name.includes(".csv"))
console.log(csvFiles);
}
- Once we have the files then get the dates from the file names
name: '03-13-2020.csv'
- Parse each CSV
import fetch from "isomorphic-unfetch";
import parse from "date-fns/parse";
import parseISO from 'date-fns/parseISO';
import { isValid } from "date-fns";
interface Entry {
province: string;
country: string;
lastUpdate: Date | null;
confirmed: number;
deaths: number;
recovered: number;
long: string | null;
lat: string | null;
}
enum CSV {
PROVIENCE,
COUNTRY,
LAST_UPDATE,
CONFIRMED,
DEATHS,
RECOVERED,
LAT,
LONG
}
function parseData(data: string) {
const lines = data.split("\n");
// Removes the CSV Headers
lines.shift();
const entries: Entry[] = [];
lines.forEach((line: string) => {
const items = line.trim().split(",");
if (items.length >= 6) {
entries.push({
province: items[CSV.PROVIENCE],
country: items[CSV.COUNTRY],
lastUpdate: parseDate(items[CSV.LAST_UPDATE]),
confirmed: items[CSV.CONFIRMED] ? parseInt(items[CSV.CONFIRMED]) : 0,
deaths: items[CSV.DEATHS] ? parseInt(items[CSV.DEATHS]) : 0,
recovered: items[CSV.RECOVERED] ? parseInt(items[CSV.RECOVERED]) : 0,
lat: items[CSV.LAT] ? items[CSV.LAT] : null,
long: items[CSV.LONG] ? items[CSV.LAT] : null,
});
}
});
return entries;
}
async function fetchCSV(filename: string) {
const response = await fetch(
`https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_daily_reports/${filename}.csv`
);
const data = await response.text();
const entries = parseData(data);
}
from predictcovid.
Very quick and hacky, let me know your thoughts
from predictcovid.
@kn-neeraj: Please see the above comment. A pull request to switch out data source from scraping Worldometers to importing from https://github.com/CSSEGISandData/COVID-19 would enable India support (and would be a very welcome update, as it would enable support for every other country too).
@williamluke4 / @kn-neeraj: The file that needs to be rewritten to pull from this new source is https://github.com/lachlanjc/covid19/blob/master/api/src/functions/scrape.js. Once that is rewritten, I will set up a separate service to call that function every hour so the site is constantly updated.
from predictcovid.
Are there any tasks to tackle? I'd love to help out!
from predictcovid.
from predictcovid.
@rishiosaur Check out #4, where I've adapted what @jajoosam started. It has some remaining issues (UK & Netherlands data primarily, haven't investigated why those seem broken), then the primary task is wiring up the data being fetched to saving to the database.
from predictcovid.
@lachlanjc line 68 of new scrape.js should be
let dates = Object.keys(agg).filter(
, newscrape.js
should work then - without it there'd be an issue with every country with multiple COVID-19 documented regions :)
from predictcovid.
Related Issues (5)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from predictcovid.