rootsdev / genscrape Goto Github PK
View Code? Open in Web Editor NEWJavaScript library that aids in scraping person data off of genealogy websites
License: MIT License
JavaScript library that aids in scraping person data off of genealogy websites
License: MIT License
Will need to update existing source references (or add new ones) to correctly point to the source descriptions.
Examples for testing:
It's mostly used for forEach
, isArray
, isFunction
, find
. It ought to be easy to remove.
Two choices:
I could be useful to state that the data was assembled by genscrape and even specify which version of genscrape. Does the GedcomX model allow for that?
http://www.findagrave.com/cgi-bin/fg.cgi?page=sh&GRid=92732955&
http://www.findagrave.com/cgi-bin/fg.cgi?page=gr&GRid=92732955&
Those two pages are the same profile. The only difference is that the sh
page has a banner that says "You are taking a random walk through our online cemetery." The Find A Grave scraper currently only works on the gr
pages.
We used to include the version number in the file name but CDNs expect the file name to be the same every time (rightly so) so we stopped doing that. Instead lets add the version number in a comment at the beginning of the package files.
Get ready for the new version of Ancestry
We're not looking for the gender yet. We'll need to do that eventually. Before 1.0.
Document how to write it, how to add it to the genscrape lib, and how to test it.
Only FS uses them. We don't need them. They just bloat the data.
We can get rid of them easily by removing links
from GedcomX.ExtensibleData.jsonProps
While testing with WikiTree X and RootsSearch I noticed that the Person/birthDate value isn't recognized/handled. The scraper is similar tot the Genealogie Online scraper which does handle the person/birthDate value. Microdata on both sites are the same (and valid).
There are two general usecases:
genscrape supports usecase (1) right now but doesn't support (2).
Should we add a configuration option that only allows one event to be fired or should we add a destroy() method that apps are expected to call when they want usecase (2)? Perhaps both.
We know that the first name will be the BirthName and all others will be AlsoKnownAs
Why does the urlPatternToRegex
test fail in node v4 and up?
All files currently in src
except for main.js
will be moved into it. That will make the code a little more organized.
As we work to rewrite all parsers for the schema upgrade, let's remove the dependency on jQuery.
With blank first name and last name known, the last name got incorrectly populated into the first name field on Wikitree.com
With nickname in "Quotes" it was ignored and not populated into the nicknames field.
As we work on #23, we are adding citations via a SourceDescription
. They're not very standard right now. Once it ships, lets revisit citations including exploration of citation templates.
Test with http://person.ancestry.com/tree/34211057/person/19227526747/facts
Family names are parsed incorrectly. HTML might have changed.
A user reported that this is happening.
Apps working with genscrape might want to programmatically detect and display information about the site that the data originates from. They shouldn't have to generate their own URL matching algorithm for all the different websites because we already do that.
SourceDescriptions have a repository
property which is a URI that resolves to an Agent which can be used to describe an owner of a repository.
TODO:
Make sure we handle UTF8.
Test: https://familysearch.org/tree/#view=ancestor&person=L5CB-71D§ion=details
Current:
Find A Grave, database and images (http://findagrave.com : accessed 2 February 2017), memorial page for Lucy Brancheau Loranger (1824 - 1870) - Find A Grave Memorial.
Requested:
Find A Grave, database and images (http://findagrave.com : accessed 2 February 2017), memorial #93209858 for Lucy Brancheau Loranger (1824 - 1870) - Find A Grave Memorial.
For example, FamilySearch Family Tree person ID or Find A Grave memorial number.
We will set the persons' IDs to be the ID we want to expose (instead of the auto-incrementing IDs we've been using).
Related to #33
Find A Grave recently enabled and forwards all traffic to https.
genscrape/src/scrapers/findagrave.js
Line 6 in 1af4027
In working on #33 I see that we need to be more deliberate about where we attach SourceDescriptions. FamilySearch generates SourceDescriptions on their own and have them attached to the root GEDCOM X element. For all other sites we have been generating one SourceDescription and attaching it to all persons and relationships in the document. We did that before realizing that the root element had a description
property.
I believe we should use the root level description
property. But should we continue attaching that same SourceDescription to all persons and relationships?
Internally, genscrape only has a list of regexes for matching against urls. To make this work we would need to add another parameter in the register
function that accepts an id/name.
genscrape uses the gensearch schema. It's a simple schema designed for searching on genealogy websites. But genscrape has many more use cases and ought have a more advanced schema with first-class support for multiple persons, relationships, and sources.
{
"persons": [],
"relationships": [],
"sources": []
}
That looks very similar to the GEDCOM X JSON format, with the only exception being that GEDCOM X has sourceDescriptions
instead of sources
. Despite how much I dislike the term sourceDescriptions
and it's schema, I _really_ like the idea of not having to create my own data format.
I figure we'll use PhantomJS for testing if since we need to simulate a browser.
Authentication will be an issue since most trees and records are behind a paywall, or at least require a login. We can't put credentials in this public repo so we'll probably need a script of some sort that prompts the developer for relevant auth credentials before running the test suite.
Figure out why it seems to be broken
Tree:
.co.uk
.ca
.com.au
.de
.it
.fr
.se
.mx
Records:
.co.uk
.ca
.com.au
.de
.it
.fr
.se
.mx
Or perhaps just the English ones for now because the Ancestry parser only works with English labels.
We are currently committing the built genscrape.js
file. Let's stop doing that.
They are severely out of date, though we might want to wait until instabul has a final 1.0 release.
https://david-dm.org/rootsdev/genscrape#info=devDependencies&view=table
Using a common schema some will inherently lead to some data loss. Add an option to get the complete source-specific schema.
Another thought is to use the source schema by default and have an option for converting it into a shared schema.
This library will just be a utility. It is designed to function multiple environments such as a browser extension or a node.js app. We will need to devise an API that functions well for common use cases.
It needs to be async since websites such as FamilySearch and MyHeritage load data via AJAX on some pages.
Also, in an environment like a browser extension where a user might navigate to multiple pages, there will be more than one data
or load
event and might even be a no-data
event. This is just begging for an event driven system. Perhaps something like node's common method of on('event_name', function(data){ })
.
Example: http://person.ancestry.com/tree/35219398/person/18763680718/facts
The data is loaded but scraping returns nothing.
Get sources attached to an ancestry tree profile.
It would be nice if it pulled the Marriage date and location when present in a profile from Ancestry.com
Yeah, I know, it's a very niche website, but hey! Free open records API! May as well include it too.
Full docs here:
http://docs.geshergalicia.apiary.io/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.