feddelegrand7 / ralger

ralger makes it easy to scrape a website. Built on the shoulders of titans: rvest, xml2.

Azzul Fellawen 👋

I'm Fodil, a PhD in Economics and an avid R and Shiny developer.

Here a list packages I've recently developed:

R

---

• 📦 {ralger}
• 📦 {batata}
• 📦 {pivta}
• 📦 {fabricerin}
• 📦 {rintimg}
• 📦 {flipdownr}
• 📦 {radous}
• 📦 {savonliquide}
• 📦 {cronologia}
• 📦 {ddplot}

VSCode

---

{shinysnip} a VSCode extension that allows you to generate Shiny code snippets.

Tech

ralger's Issues

incorect link in README

The link from README used for my_link:

http://www.shanghairanking.com/ARWU2020.html

is probably incorrect. Should it point to

http://www.shanghairanking.com/rankings/arwu/2021

instead

Is it possible to download data from market.cboe.com with ralger?

Hi,

ralger looks very promising.

I was wondering whether there is any way to download data from http://markets.cboe.com/us/futures/market_statistics/historical_data/ using ralger?

I mean, let's say I want to download all links if I select "VX+VXT- Cboe Volatility Index (VX) Futures, Includes TAS" from the dropdown. I can then see a list of links grouped by year. Selecting first one really downloads the file if I do this in browser. However if I copy the link and use read.csv("http://markets.cboe.com/us/futures/market_statistics/historical_data/products/csv/VX/2021-01-20/CFE_VX_F1.csv") it fails. Is there a trick in ralger that can accomplish successful download?

> data <- read.csv("http://markets.cboe.com/us/futures/market_statistics/historical_data/products/csv/VX/2021-01-20/CFE_VX_F1.csv")
Error in file(file, "rt") : 
  cannot open the connection to 'http://markets.cboe.com/us/futures/market_statistics/historical_data/products/csv/VX/2021-01-20/CFE_VX_F1.csv'
In addition: Warning message:
In file(file, "rt") :
  cannot open URL 'http://markets.cboe.com/us/futures/market_statistics/historical_data/products/csv/VX/2021-01-20/CFE_VX_F1.csv': HTTP status was '404 Not Found'

Any other suggestions welcome (Python or whatever).

Thanks.

Writing more documentation

need to write some docs for the following functions title_scrap, paragraph_scrap, weblink_scrap within the README.Rmd. Also, need to update the description within the DESCRIPTION file.

Extract Href Attributes from Given Nodes

How can I extract the href attributes from given nodes? Is there a ralger alternative for html_attr() of rvest?

weblink_scrap() extracts all links and in some cases string filtering parameter may not be enough.

Adding more Web Scraping Shortcuts

need to look for more complex web scraping procedures that can be simplified.

Error: Table has inconsistent number of columns. Do you want fill = TRUE?

I tried to use {ralger} to reproduce what was done at https://www.r-bloggers.com/scrape-html-table-using-rvest/.

library(ralger)
library(tidyverse)

df <- table_scrap("https://en.wikipedia.org/wiki/List_of_highest-grossing_films_in_the_United_States_and_Canada", choose = 1)

ralger::table_scrap() threw this Error: Table has inconsistent number of columns. Do you want fill = TRUE? with no useful guide to solve this for a novice.

I tried to add fill = TRUE in

table_scrap <- function (link, choose = 1, header = T, askRobot = FALSE) 
{
  if (askRobot) {
    if (paths_allowed(link) == TRUE) {
      message(green("It's ok you're allowed to scrap this web page"))
    }
    else {
      message(bgRed("WARNING: you're not allowed to scrap this web page"))
    }
  }
  table <- link %>% read_html() %>% html_table(header, fill = TRUE) %>% 
    purrr::pluck(choose)
  return(table)
}

and it works

table_scrap("https://en.wikipedia.org/wiki/List_of_highest-grossing_films_in_the_United_States_and_Canada")

Error: ' http://elmouchir.caci.dz/search_results.php?keyword=&category=&location=Alger&submit=Find ' does not exist in current working directory

i got this error...any help?