Ruby gem that parse HTML containing microformats/microformats2 and returns Ruby objects, a Ruby hash or a JSON hash
Home Page: https://rubygems.org/gems/microformats
License: Creative Commons Zero v1.0 Universal
Add deprecation warnings and forward compatible versions of functions so users can upgrade their function calls early.
http://microformats.org/wiki/microformats-2#h-card_org_h-card
a microformat that are not attached to a specific property:
<div class="h-card">
<a class="p-name u-url"
href="http://blog.lizardwrangler.com/"
>Mitchell Baker</a>
(<a class="h-card"
href="http://mozilla.org/"
>Mozilla Foundation</a>)
</div>{
"items": [{
"type": ["h-card"],
"properties": {
"name": ["Mitchell Baker"],
"url": ["http://blog.lizardwrangler.com/"]
},
"children": [{
"type": ["h-card"],
"properties": {
"name": ["Mozilla Foundation"],
"url": ["http://mozilla.org/"]
}
}]
}]
}php-mf2 tests are comprehensive https://github.com/indieweb/php-mf2/blob/master/tests/Mf2/URLTest.php
Implementation of parsing rels: #7
Parsing spec: http://microformats.org/wiki/microformats2-parsing#parse_a_hyperlink_element_for_rel_microformats
I've committed a failing test, but I really don't know enough about this stuff to propose a solution. It blows up on this line. Is the markup in the test valid? I got it from here in g5-hub.
See
microformats/microformats2-parsing#2 (comment)
for the proposal.
This must be behind an additional flag to the parser for 'experimental'
title pretty much says it all, parsing ends up having a bunch of
`
"https://example.com/aaronpk": {
"rels": [
]
},
`
but the URLs are not in the 'rels' section at all.
I believe it should not be in either
I don't think this case works right now.
<!DOCTYPE html>
<html>
<body>
<div class="h-card">
<div class="p-adr h-adr">
<span class="p-street-address">
123 Rainbow Lane
</span>
</a>
</div>
</div>
</body>
</html>I'm not sure how to use it, it seems to have all the data parsed, because I can to .to_json and it looks right but I can't get the data out. I am trying to get the author name, his/her url, the content and the h-entry url, so nothing fancy.
I am trying to do it with: https://brid-gy.appspot.com/like/twitter/jeena/424554756917702656/109427493
I know it should be possible because this looks exactly like I want to have it: http://indiewebify.me/validate-h-entry/?url=https%3A%2F%2Fbrid-gy.appspot.com%2Flike%2Ftwitter%2Fjeena%2F424554756917702656%2F109427493
collection.entry.content.to_s works fine, I am getting "favorited this." and so does collection.entry.url.to_s. But when I try to get the author, then I get R Ç tag:twitter.com,2013:schnarfedwhich is understandable, it is the u-id, but then I can't get the collection.entry.author.name.to_s because I get:
NoMethodError: undefined method `name' for #<Microformats2::Property::Text:0x007fbbfc88ac60>
same with collection.entry.author.url.to_s. So Either I don't understand how this should be called or something is not working. Could someone help me out here?
HTML:
<div class="h-entry">
<time class="dt-published" value="2018-01-12">January 12</time>
</div>
Expected result:
{
"items": [
{
"type": [
"h-entry"
],
"properties": {
"published": [
"2018-01-12"
],
"name": [
"January 12"
]
}
}
],
"rels": {
},
"rel-urls": {
}
}
Current result:
{
"items": [
{
"type": [
"h-entry"
],
"properties": {
"published": [
"2018-01-12 00:00:00"
],
"name": [
"January 12"
]
}
}
],
"rels": {
},
"rel-urls": {
}
}
http://microformats.org/wiki/value-class-pattern#Date_and_time_parsing
http://microformats.org/wiki/value-class-pattern##If+by+parsing+the+%22value%22+element(s)
microformats gemThe name of the Microformats Ruby Parser is changing from "microformats2" to "microformats". This is a one time change. (Thanks to @chrisjpowers for transferring the namespace to us.)
Follow these instructions to migrate.
1. Install the new gem. Uninstall the old gem.
gem install microformats
gem uninstall microformats2
2. Change any Gemfiles from:
gem "microformats2"
to
gem "microformats"
bundle
3. Change any requires from:
require "microformats2"
to
require "microformats"
3.1microformats2 to microformats4.0 of microformats (exact same code base as 3.1, except for post install note)Based on some of the conversation in #84…
Which versions of Ruby should we continue supporting?
Some data:
Some related questions:
NOT HIGH PRIORITY
http://microformats.org/wiki/microformats-2#VENDOR_EXTENSIONS
would be cool if something special happened
It seems the parser is not handling nested objects properly.
For example, this URL: http://aaronparecki.com/notes/2014/07/04/2/indiewebcamp-latergram
It appears the comment authors and comment URLs all show up under the main h-entry when in reality they should be under children of the main h-entry as their own h-cite objects.
Compare the result of the PHP parser
Had the idea of parsing a page and pulling out only a specific comment and how would that work (assuming it isn't posted from somewhere else). The idea would be to give a URL that has a fragment and the result items would contain anything from that id and below.
Would have to look at how this would work exactly, would likely need the whole page for rels and base and such.
As part of #77, I originally introduced a logo for microformats-ruby. It's a derivation of the official logo mark (found on the Spread Microformats wiki page):
Conceptually, this design marries the microformats logo mark with a color scheme similar to that found throughout the Ruby community. Basically: everything's red.
In the course of discussion on #77 (and in IRC/Slack), some licensing issues were brought up. Turns out those aren't an issue as the original author of the logo mark has dedicated it to the public domain, pursuant to the microformats wiki's licensing.
I'm opening this issue to track conversation around the logo:
README.md as I initially did?I am having problems with what I get from the parser, it looks like there is some kind of a bug or something. I am parsing this website: http://snarfed.org/2014-02-03_re-barnaby-walters-latest-php-library-extracted-from-taproo and when you use the PHP parser mf2 it seems to parse the website like expected: http://pin13.net/mf2/?url=http://snarfed.org/2014-02-03_re-barnaby-walters-latest-php-library-extracted-from-taproo but with G5/microformats2 I get this JSON: https://jeena.net/t/mf2.json
You can see that h-entry here has two URLs:
"url": [
"http://snarfed.org/",
"http://snarfed.org/2014-02-03_re-barnaby-walters-latest-php-library-extracted-from-taproo"
]
even if the first URL shouldn't be there because it belongs either to the h-card at the top or to to the author h-card in the middle.
The microformats parsing spec was updated yesterday to reflect a 2015 resolution on <link> elements. The Ruby parser should be updated to reflect this.
An HTML document that uses an existing rel-canonical <link> element to communicate the page’s u-url and p-name:
<!doctype html>
<html class="h-entry">
<head>
<link rel="canonical" class="u-url p-name" href="https://example.com/" title="Example.com homepage">
</head>
<body></body>
</html>Expected parser output (rels and rel-urls have been left out):
{
"items": [
{
"type": [ "h-entry" ],
"properties": {
"name": [ "Example.com homepage" ],
"url": [ "https://example.com/" ]
}
}
]
}u-url and u-author on the <link> elements with rel="canonical" and rel="author" respectively.version 4.0.8 was able to be released and passed all CI tests, but it failed when testing on a different system as 'require "set"' was not in the code base... why do all tests pass despite this??
undefined method `url' for #<Microformats::ParserResult:0x00005556e27ddaa0>
when accessing mf2.entry.comment.author.url in 4.0.9. This is a breaking change from 4.0.7 where it worked fine.
<article class="h-entry"> <data class="p-content" value="">This should not be the value of the content attribute</data></article>
should have content=[""]
its not returning content at all
via @aaronpk
Parsing spec: http://microformats.org/wiki/microformats2-parsing#parse_a_hyperlink_element_for_rel_microformats
http://microformats.org/wiki/rel
http://diveinto.html5doctor.com/semantics.html#link
http://en.wikipedia.org/wiki/XFN
http://microformats.org/wiki/RelMeAuth
https://github.com/aaronpk/IndieAuth
http://aaronparecki.com/
The parser does not seem to implement implied properties correctly.
When parsing https://commentpara.de/comment/43.htm it does not fill the author name property with its value:
$ ./tmp/ruby-microformats2-test.rb https://commentpara.de/comment/43.htm
{
"items": [
{
"type": [
"h-entry"
],
"properties": {
"author": [
{
"value": "Anonymous",
"type": [
"h-card"
],
"properties": {
"photo": [
"https://commentpara.de/img/anonymous.svg"
],
"name": [
""
],
"url": [
"https://commentpara.de/user/3.htm"
]
}
}
],
"url": [
"https://commentpara.de/comment/43.htm"
],
"content": [
"Digikam is able to auto-tag images with the people that are on the photo. This already exists and is open source."
],
"name": [
"Digikam is able to auto-tag images with the people that are on the photo. This already exists and is open source."
]
}
}
],
"rels": {
"in-reply-to": [
"https://jeena.net/notes/754"
],
"u-in-reply-to": [
"https://jeena.net/notes/754"
]
}
}
not properly removing style tags.
the tag is removed but the content of the style tag gets left in the results
For example https://travis-ci.org/indieweb/microformats2-ruby/builds/150143352
Gem::InstallError: ruby_dep requires Ruby version >= 2.2.5, ~> 2.2.
An error occurred while installing ruby_dep (1.4.0), and Bundler cannot
continue.
Implementation of parsing rels: #7
Parsing spec: http://microformats.org/wiki/microformats2-parsing#parse_a_hyperlink_element_for_rel_microformats
there is a test in the test suite about this, it is being set marked pending for now
this includes some parsing changes and other issues such as #86
When parsing a document, all URLs returned should be absolute URLs based off of the URL that is being parsed.
If running Microformats2.parse with a URL as an argument, just use that as the URL to compute relative URLs from. Otherwise, if parsing an HTML string directly, need an additional parameter to pass the URL into.
Now that this gem's test suite is back in sync with microformats/tests, would it be possible to directly include the code in that repo into this project via a git submodule or some mechanism other than copy/paste?
Git submodules aren't without their overhead, but we could mitigate potential problems by updating some documentation with details on how to handle initial project check out, updating the test suite version, etc.
In order to report coverage to Code Climate, someone with access to this project's Code Climate and Travis CI configuration will need to add a CC_TEST_REPORTER_ID environment variable. To do that…
https://codeclimate.com/repos/<some-long-string>/edit.CC_TEST_REPORTER_ID for the name and paste in the value grabbed from Code Climate in step 2.
getting a code climate score of 0.7, which is pretty abysmal.
While i don't agree with all of the rules they use, there is a lot of work that can be done to improve the readability of the code base.
Similar to microformats/mf2py#51 and microformats/php-mf2#69, the Ruby parser is stripping <br> tags rather than converting them to newlines when converting HTML to plaintext.
This is very apparent on Tantek's autoformatted posts. Compare the name and content.value for one of his posts:
It'd be nice if there was a cli tool that I could pass an URL, and it would dump the extracted microformats.
Goal would be to have tool to quickly debug potential parsing problems.
This could farm out to something else.
When I install the gem like you suggest with gem install microformats2 and then run the example code from this page (I have it in test.rb) then I am just getting an error. I see that it installs version 1.0.2 instead of 2.0.0, so how can I install version 2.x?
➜ jeena@Lala gem install microformats2
Successfully installed microformats2-1.0.2
Parsing documentation for microformats2-1.0.2
Done installing documentation for microformats2 after 0 seconds
1 gem installed
➜ jeena@Lala ruby test.rb
test.rb:6:in `<main>': undefined method `card' for #<Hash:0x007fe66bb2a3a0> (NoMethodError)
require 'microformats'
doc = Microformats.parse("http://tantek.com")
puts JSON.pretty_generate(doc.to_h)
This worked before. Something changed / broke.
TypeError: no implicit conversion of nil into String
method value= in parser.rb at line 35
method block in parse in parser.rb at line 35
method traverse in node.rb at line 585
method block in traverse in node.rb at line 584
method block in each in node_set.rb at line 187
method upto in node_set.rb at line 186
method each in node_set.rb at line 186
method traverse in node.rb at line 584
method block in traverse in node.rb at line 584
method block in each in node_set.rb at line 187
method upto in node_set.rb at line 186
method each in node_set.rb at line 186
method traverse in node.rb at line 584
method block in traverse in node.rb at line 584
method block in each in node_set.rb at line 187
method upto in node_set.rb at line 186
method each in node_set.rb at line 186
method traverse in node.rb at line 584
method block in traverse in node.rb at line 584
method block in each in node_set.rb at line 187
method upto in node_set.rb at line 186
method each in node_set.rb at line 186
method traverse in node.rb at line 584
method block in traverse in node.rb at line 584
method block in each in node_set.rb at line 187
method upto in node_set.rb at line 186
method each in node_set.rb at line 186
method traverse in node.rb at line 584
method parse in parser.rb at line 28
method parse in microformats.rb at line 19
method <main> in mf.rb at line 3
See microformats/microformats2-parsing#6 for details
(See also microformats/php-mf2#126 for the same bug in php-mf2)
The following snippet has date, time and timezone offset in three different elements, using the value-class-pattern.
<div class="h-event">
<span class="e-summary">HomebrewWebsiteClub Berlin</span> will be next on
<span class="dt-start">
<span class="value">2017-05-31</span>, from
<span class="value">19:00</span> (UTC<span class="value">+02:00</span>)
</span> to <span class="dt-end">21:00</span>.</div>I would have expected a value of 2017-05-31 19:00+02:00 for start, per http://microformats.org/wiki/value-class-pattern#Date_and_time_parsing
[…] the parser assembles the overall datetime value by concatenating the specific date, " " (space character) and specific time (if time was specified, with 00 minutes implied if no minutes are provided), and specific timezone (if timezone and a specific time was specified)
Instead the timezone is dropped:
"start": [ "2017-05-31 19:00" ]See: microformats/microformats-parser-website-ruby#6
I fixed it in the site by .stripping the params[:url] before parsing. But it feels like that should be in the parser itself.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.