jazilch / hubspot-web-scraper Goto Github PK

Currently when scraping a blog we assume that the images on the page will be using an img tag and not using CSS to add the image to the page.

We should add support to the scraper to allow for the background images to be scraped from the website.

We might be able to add in a dropdown to the form when they are filling out the form to select if the scraper should account for background images or not. The default should be No || false.

I've been trying to run the tool with this blog - https://www.hycu.com/blog/ (HubSpot blog - http://685119.hs-sites.com/blog)

I get to step 2 and it finds the page slugs for the posts but only the first post image appears, the rest are blank. When I click 'Next' at this step the tool fails. See video below:

https://www.loom.com/share/92c826d3a58f415c9d347d4e75ac29da

We should handle lazy loaded images as we commonly see this in Wordpress. Normally a typical schema for this would be to have a data-src attribute and one the page loads to grab the src from this attribute and set it for the src of the image.

We should add in Support for if the images are in base64 data type. Currently we won't be able to upload these types of files and the upload will fail when trying to upload these data types. We should think about converting these types to a png