JSONDecodeError within scrape() method about instascrape

[esvban]

i tried going through this with the debugger.

I don't know the original intent of this code, or how it worked before https://github.com/chris-greening/instascrape/blob/master/instascrape/core/json_algos.py#L75

but basically the json strings returned by this filter are not all pure json data, but also contain javascript code. Checking for any script tag with 'config' in it is not the correct way to find json script tags, at least anymore. I tried lookin at the source html and couldn't figure out which json data is actually needed. The media html is using a bunch of async loading of json data embedded in javascript code, as well, that will be difficult to parse correctly, probably designed to make scraping the website very difficult without using the Facebook developer API. I couldn't even figure out how to manually download a single video from a reel page (i.e. using chrome developer tools / inspector) either. I suppose long gone are the days of just searching for a ".mp4" link in the source and downloading it, with all the obfuscation and anti-scraping / anti-downloading going on...

There hasnt been any commit on this package in over a year, probably have to fix it our selves / fork it if the developers are no longer active.

edit: after a big more digging around, finally found the mp4 sources for reels in the network profiler with some url editing... The download speed is super slow, and the audio / video are split into separate files, but this is good enough for me, at least for now.

[santimorales0326]

Thank you for your quick response. Do you know how efficient scrapping with Facebook dev API is? I'd appreciate any resource that you can provide.

…

On Thu, Jul 28, 2022 at 9:43 AM Stephen Hopkins ***@***.***> wrote: i tried going through this with the debugger. I don't know the original intent of this code, or how it worked before https://github.com/chris-greening/instascrape/blob/master/instascrape/core/json_algos.py#L75 but basically the json strings returned by this filter are not all pure json data, but also contain javascript code. Checking for any script tag with 'config' in it is not the correct way to find json script tags, at least anymore. I tried lookin at the source html and couldn't figure out which json data is actually needed. The media html is using a bunch of async loading of json data embedded in javascript code, as well, that will be difficult to parse correctly, probably designed to make scraping the website very difficult without using the Facebook developer API. There hasnt been any commit on this package in over a year, probably have to fix it our selves / fork it if the developers are no longer active. — Reply to this email directly, view it on GitHub <#164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2HQJCULX3YIJRJDWOYH4UTVWI2ZPANCNFSM54VZVNIQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[esvban]

I don't know too much about the API, except it is only usable by registered and most likely required to be approved, facebook integrated applications.

[ivangoranov]

Hi there, has anybody solved this yet?

[furmanlukasz]

Hi, I'm hiting the same error. anybody have a clue how to solve it?

[nsde]

This project isn't maintained anymore ._.