website-scraper / website-scraper-existing-directory

Plugin for website-scraper which allows to save resources to existing directory

website-scraper-existing-directory

Plugin for website-scraper which allows to save resources to existing directory.

Please keep in mind that saving to existing directory may overwrite your files. Be careful with it!

Requirements

• nodejs version >= 14.14
• website-scraper version >= 5

Installation

npm install website-scraper website-scraper-existing-directory

Usage

import scrape from 'website-scraper';
import SaveToExistingDirectoryPlugin from 'website-scraper-existing-directory';

await scrape({
    urls: ['http://example.com'],
    directory: '/path/to/save',
    plugins: [ new SaveToExistingDirectoryPlugin() ]
});

website-scraper-existing-directory's Issues

Usage in electronJS throws an error which says: SaveToExistingDirectoryPlugin not a constructor

I am using this in the main.js of an electron application.
My imports look like this:
const scrape = (...args) => import('website-scraper').then(({default:scrape})=>scrape(...args)); const SaveToExistingDirectoryPlugin = (...args) => import('website-scraper-existing-directory').then(({default:SaveToExistingDirectoryPlugin})=>SaveToExistingDirectoryPlugin(...args));
My scrape options like this:
let options = { urls: [url], directory: cache_dir, plugins: [ new SaveToExistingDirectoryPlugin() ] }
Then I execute scrape with the options:
scrape(options).then(...)
This throws an error which says "SaveToExistingDirectoryPlugin is not a constructor"

How to resolve this

Charset on final file is not like expected

Configuration

version: [email protected]

options: [provide your full options object]

const options = {
  urls: ['https://www.appmasters.io/pt/'],
  directory: '/Users/tiagogouvea/www/scrapper/test/app/',
  plugins: [ new SaveToExistingDirectoryPlugin() ]
};

Description

Trying to scrap a website that has <meta charSet="utf-8"/> results on a content without correct charset presentation.

Following is a screenshot showing the original content and the scrapped one.

Expected behavior: That the save filed was on the same charset.

Actual behavior: It failed to save on the current way.

Additional Information

Congratulations for this amazing project! 👏👏👏