Easier way to scrape regular html data to data frame about rvest

[hadley]

Alternatively, you could write it like this:

nodes <- "http://pyvideo.org/category/50/pycon-us-2014" %>%
  html %>%
  html_nodes("div.video-summary-data")

column <- function(x) nodes %>% html_node(xpath = x) %>% html_text()

df <- data.frame(
  title = column("div[1]//a"),
  author = column("div[3]//a"),
  date = column("div[4]//small[1]"),
  language = column("div[4]//small[2]"),
  description = column("div[5]//p"),
  stringsAsFactors = FALSE
)

I think this is simple enough that you don't need a wrapper

[abresler]

Man this is amazing!! Soon people will see how powerful r can be for webscraping

[hadley]

I like it - I need to think through the syntax a bit though. It's a bit too reliant on xpath currently - i.e. there's no way to access an attribute with a css selector. It'd also be nice if it worked in a where you could easily test each component, before building up into a data frame.

(Also I think this will require a variant of html_node() that doesn't unlist(), and assumes the selector only pulls out one node. Now that I think of it, html_node() is basically [, and we need an equivalent of [[. Maybe html_node() should be renamed to html_nodes() and html_node() should only ever pull out one node per input element.)

[hadley]

Another idea is to implement the css selector extensions used by https://github.com/EricChiang/pup

[renkun-ken]

Great thinking! The design of html_node() and html_nodes() works with many problematic cases in which the nodes are not quite regular. The code above does not really work due to some exceptional cases in the webpage, but with these two functions, it works correctly without too much worry about the missing/blank values in some fields.

[renkun-ken]

Using html_node() and html_nodes(), the code can be much less error-prone.

library(rvest)

nodes <- "http://pyvideo.org/category/50/pycon-us-2014" %>%
  html %>%
  html_nodes("div.video-summary-data")

cols <- list(
  title = "div[1]//a//text()",
  author = "div[3]//a//text()",
  date = "div[4]//small[1]//text()",
  language = "div[4]//small[2]//text()",
  description = "div[5]//p//text()")

df <- cols %>%
  lapply(function(col) {
    nodes %>%
      html_node(xpath = col) %>%
      html_text
  }) %>%
  data.frame(stringsAsFactors = FALSE)

[renkun-ken]

@hadley, fantastic use, just forget about using function :) This workflow looks very natural now.