Authentication issues using get_league and get_league_entries about fplscrapr

[wiscostret]

Hi folks, this took way too long to get back to, and I apologize. It's a doozy - it turns out the FPL API no longer requires authentication to get the league data. And I hadn't realised. I will be updating the package shortly, but essentially you can just cut out the curl authentication bits of the get_league_*() commands, e.g. as follows:

get_league <- function(leagueid = NULL, leaguetype = "classic"){
  if(is.null(leagueid)) stop("You'll need to input a league ID, mate.")
  if(length(leagueid) != 1) stop("One league at a time, please.")
    {
    league <- jsonlite::fromJSON(paste("https://fantasy.premierleague.com/api/leagues-",leaguetype,"/",leagueid,"/standings/",sep=""))
    return(league)
  }
}

get_league(45536)

I will leave this open for a bit to see if anyone still has issued, otherwise I will consider it closed with the updated package version.

[wiscostret]

Thanks for logging this. The 405 error seems to indicate that it’s an issue with the curl interfacing embedded in the get_league_ functions, but I wasn’t able to reproduce your issue, and I haven’t run in to this before, so I am admittedly guessing a bit here.

First, what client are you running this from - RStudio? version?

Second, I'm assuming from your headline you run into the same error both with get_league() and get_league_entries() - correct?

Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot.

library(fplscrapR)

fplfetchhandle <- curl::new_handle(verbose=TRUE)

curl::handle_setform(fplfetchhandle,
  login=readline("Please enter your FPL login email: "),
  password=getPass::getPass(msg="Please enter your FPL password:"),
  redirect_uri="https://fantasy.premierleague.com/a/login",
  app="plfpl-web")

fplfetchmemory <- curl::curl_fetch_memory("https://users.premierleague.com/accounts/login/", handle = fplfetchhandle)

jsonlite::fromJSON(curl::curl("https://fantasy.premierleague.com/api/leagues-classic/45536/standings/", handle = fplfetchhandle))

[lvtwoodpecker]

having this same issue. The get_league_entries function only prompted a Password input but no field for me to put in the email. Only after I ran get_league_entries the second time did it prompt the email field in the console!

Running the curl call as above worked though.

[woyzeck1978]

Thanks for logging this. The 405 error seems to indicate that it’s an issue with the curl interfacing embedded in the get_league_ functions, but I wasn’t able to reproduce your issue, and I haven’t run in to this before, so I am admittedly guessing a bit here.

First, what client are you running this from - RStudio? version?

Second, I'm assuming from your headline you run into the same error both with get_league() and get_league_entries() - correct?

Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot.

library(fplscrapR)

fplfetchhandle <- curl::new_handle(verbose=TRUE)

curl::handle_setform(fplfetchhandle,
  login=readline("Please enter your FPL login email: "),
  password=getPass::getPass(msg="Please enter your FPL password:"),
  redirect_uri="https://fantasy.premierleague.com/a/login",
  app="plfpl-web")

fplfetchmemory <- curl::curl_fetch_memory("https://users.premierleague.com/accounts/login/", handle = fplfetchhandle)

jsonlite::fromJSON(curl::curl("https://fantasy.premierleague.com/api/leagues-classic/45536/standings/", handle = fplfetchhandle))

thanks so much for your response and apologies for the late reply.

Re 1) I run the most recent R Studio version (2021.09.0 -"Ghost Orchid" Release (077589bc, 2021-09-20) for macOS
Mozilla/5.0 (Macintosh; Intel Mac OS X 12_1_0) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.10 Chrome/69.0.3497.128 Safari/537.36)

Re 2) yes, I receive the same error message with both functions.

Re 3) I've no problems logging in when running the verbose option:
$url
[1] "https://fantasy.premierleague.com/a/login?state=success"
$status_code
[1] 200

[py9mrg]

Hello,

Just to add that I've come across the same issue (both Windows through work proxy and personal Linux without proxy). Both are latest RStudio preview version, latest R, everything updated. In both cases the longer form above worked fine. Strange?

Edit: BUT, if I replace the league id in the example (45536) with the league of interest, then I get the same error. Weirdly, I can get rid of the error by replacing -classic wit -h2h but then I get a different league to intended so it seems something to do with league type, but that could be a red herring.

Edit 2 (sorry!!!): ok so I think I know the problem. It seems like the league ids have changed and/or something about how they're accessed. Using the "old" league id I've always used doesn't work any more. If you log in to the website, go to the league of interest, and check the number (https://https://fantasy.premierleague.com/leagues/[HERE]/standings/c) then using that number works fine again. I suspect the issue is all of us using outdated league ids.

[woyzeck1978]

• We are completely uploaded and fine
  < HTTP/2 405
  < server: nginx/1.19.10
  < content-type: application/json
  < allow: GET, HEAD, OPTIONS
  < x-frame-options: DENY
  < x-content-type-options: nosniff
  < referrer-policy: same-origin
  < cache-control: max-age=0, no-cache, no-store, must-revalidate, private
  < via: 1.1 google, 1.1 varnish
  < accept-ranges: bytes
  < date: Tue, 28 Dec 2021 19:27:29 GMT
  < x-served-by: cache-ams21054-AMS
  < x-cache: MISS
  Error in open.connection(con, "rb") : HTTP error 405.
  < x-cache-hits: 0
  < x-timer: S1640719649.186134,VS0,VE22
  < vary: Cookie
  < content-length: 41
  <
• Connection #0 to host fantasy.premierleague.com left intact

• We are completely uploaded and fine
  < HTTP/2 405
  < server: nginx/1.19.10
  < content-type: application/json
  < allow: GET, HEAD, OPTIONS
  < x-frame-options: DENY
  < x-content-type-options: nosniff
  < referrer-policy: same-origin
  < cache-control: max-age=0, no-cache, no-store, must-revalidate, private
  < via: 1.1 google, 1.1 varnish
  < accept-ranges: bytes
  < date: Tue, 28 Dec 2021 19:27:29 GMT
  < x-served-by: cache-ams21054-AMS
  < x-cache: MISS
  Error in open.connection(con, "rb") : HTTP error 405.
  < x-cache-hits: 0
  < x-timer: S1640719649.186134,VS0,VE22
  < vary: Cookie
  < content-length: 41
  <
• Connection #0 to host fantasy.premierleague.com left intact

Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot.

Re: point 3.

"Third, one way to troubleshoot is for you to break down the function steps, and use the 'verbose = TRUE' option for the handle to see if we can pinpoint the error source. Try running the following. This will give you more detailed information on the operations of the handle, which may help troubleshoot."

The troubleshoot info:

• We are completely uploaded and fine
  < HTTP/2 405
  < server: nginx/1.19.10
  < content-type: application/json
  < allow: GET, HEAD, OPTIONS
  < x-frame-options: DENY
  < x-content-type-options: nosniff
  < referrer-policy: same-origin
  < cache-control: max-age=0, no-cache, no-store, must-revalidate, private
  < via: 1.1 google, 1.1 varnish
  < accept-ranges: bytes
  < date: Tue, 28 Dec 2021 19:27:29 GMT
  < x-served-by: cache-ams21054-AMS
  < x-cache: MISS
  Error in open.connection(con, "rb") : HTTP error 405.
  < x-cache-hits: 0
  < x-timer: S1640719649.186134,VS0,VE22
  < vary: Cookie
  < content-length: 41
  <
• Connection #0 to host fantasy.premierleague.com left intact

[psydroid49]

Hello,

edit***

my problem is when I'm trying to fetch get_league and get_league_entries.
Once I'm prompted to login, I enter my e-mail adress and password but it keeps saying that the authentication failed.

I've tried the following:

• logged into fpl page using the same login and password - works fine
• tried to sign out from all devices before asking for league_entry data - still authentication issue
• tried to put my e-mail in quotes "" - no good
• changed my password - also does not work
• I've used another wifi, also doesn't work

I've seen that I should troubleshoot with fplfetchhandle and verbose=T. Here's the output:
`* Found bundle for host users.premierleague.com: 0x16d085f0 [can pipeline]

• Re-using existing connection! (#0) with host users.premierleague.com
• Connected to users.premierleague.com (151.101.14.133) port 443 (#0)

POST /accounts/login/ HTTP/1.1
Host: users.premierleague.com
User-Agent: R (3.6.1 x86_64-w64-mingw32 x86_64 mingw32)
Accept: /
Accept-Encoding: deflate, gzip
Content-Length: 515
Content-Type: multipart/form-data; boundary=------------------------d22dcb052c5a881d

• We are completely uploaded and fine
  <
  HTTP/1.1 403 Forbidden
  < Connection: keep-alive
  < Content-Length: 474
  < Server: nginx/1.18.0
  < Content-Type: text/html;charset=utf-8
  < X-DataDome: protected
  < Charset: utf-8
  < Cache-Control: max-age=0, private, no-cache, no-store, must-revalidate
  < Pragma: no-cache
  < Access-Control-Allow-Credentials: true
  < Access-Control-Expose-Headers: x-dd-b, x-set-cookie
  < Access-Control-Allow-Origin: *
  < X-DataDome-CID: AHrlqAAAAAMA6c0fvLg4-PYAViAisg==
• Added cookie datadome="ZghBdV5e7XJy_OFXsr-rxLUHwnzi_pIR3_COzuYZKDOfEHlvYRlfrbLJsUPqZPj73nO-vMl9k~~ODEyqM2-DAydU6cWkug29waiDzJCcNyJeMAMwIHXyItvkCeOfe" for domain premierleague.com, path /, expire 1677608107
  < Set-Cookie: datadome=ZghBdV5e7XJy_OFXsr-rxLUHwnzi_pIR3_COzuYZKDOfEHlvYRlfrbLJsUPqZPj73nO-vMl9k~~ODEyqM2-DAydU6cWkug29waiDzJCcNyJeMAMwIHXyItvkCeOfe; Max-Age=31536000; Domain=.premierleague.com; Path=/; Secure; SameSite=Lax
  < Via: 1.1 google, 1.1 varnish
  < Accept-Ranges: bytes
  < Date: Mon, 28 Feb 2022 18:15:06 GMT
  < X-Served-By: cache-fra19152-FRA
  < X-Cache: MISS
  < X-Cache-Hits: 0
  <
• Connection #0 to host users.premierleague.com left intact`

Then, running
jsonlite::fromJSON(curl::curl("https://fantasy.premierleague.com/api/leagues-classic/314/standings/", handle = fplfetchhandle))

gets me the following output:
`* Trying 151.101.114.133...

• TCP_NODELAY set
• Connected to fantasy.premierleague.com (151.101.114.133) port 443 (#0)

POST /api/leagues-classic/25861/standings/ HTTP/1.1
Host: fantasy.premierleague.com
User-Agent: R (3.6.1 x86_64-w64-mingw32 x86_64 mingw32)
Accept: /
Accept-Encoding: deflate, gzip
Cookie: datadome=ZghBdV5e7XJy_OFXsr-rxLUHwnzi_pIR3_COzuYZKDOfEHlvYRlfrbLJsUPqZPj73nO-vMl9k~~ODEyqM2-DAydU6cWkug29waiDzJCcNyJeMAMwIHXyItvkCeOfe
Content-Length: 515
Content-Type: multipart/form-data; boundary=------------------------3fd1b4c86d7b3656

• We are completely uploaded and fine
  < HTTP/1.1 405 Method Not Allowed
  < Connection: keep-alive
  < Content-Length: 41
  < Server: nginx/1.19.10
  < Content-Type: application/json
  < Allow: GET, HEAD, OPTIONS
  < X-Frame-Options: DENY
  < X-Content-Type-Options: nosniff
  < Referrer-Policy: same-origin
  < Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
  < Via: 1.1 google, 1.1 varnish
  < Accept-Ranges: bytes
  < Date: Mon, 28 Feb 2022 18:16:41 GMT
  < X-Served-By: cache-hhn4059-HHN
  < X-Cache: MISS
  < X-Cache-Hits: 0
  < X-Timer: S1646072202.510089,VS0,VE23
  < Vary: Cookie
  <
• Connection #0 to host fantasy.premierleague.com left intact
  Error in open.connection(con, "rb") : HTTP error 405.
• Closing connection 0
• schannel: shutting down SSL/TLS connection with fantasy.premierleague.com port 443`

I've edited my previous post for it to be more readable. Was in a rush previously.
Thanks for your help!