GithubHelp home page GithubHelp logo

dstark5 / gnews-scraper Goto Github PK

View Code? Open in Web Editor NEW
7.0 2.0 3.0 157 KB

GNewsScraper is a TypeScript package that scrapes article data from Google News based on a keyword or phrase. It returns the results as an array of JSON objects, making it convenient to access and use the scraped information

Home Page: https://www.npmjs.com/package/gnews-scraper

TypeScript 100.00%
data-scraping gnews gnews-api google-news json-parsing keyword-search typescript web-automation web-crawling web-data-extraction

gnews-scraper's Introduction

GNews Scraper

GNewsScraper is a TypeScript package that scrapes article data from Google News based on a keyword or phrase. It returns the results as an array of JSON objects, making it convenient to access and use the scraped information.

MIT License Static Badge GitHub issues

Installation

 npm install gnews-scraper

Warning: This package is native ESM and no longer provides a CommonJS export. If your project uses CommonJS, you will have to convert to ESM or use the dynamic import() function. Please don't open issues for questions regarding CommonJS / ESM.

Usage

// Import the package
import GNews from 'gnews-scraper';

// Execute within an async function,
// pass a options object (further documentation below)
const news=await GNews({
    searchQuery:"Nasdaq",
    prettyUrl:true,
    cache:true,
    cacheTTL:3600,
    timeframe:'7d',
    queryParams: {
        hl:"en-US",
        gl:"US",
        ceid:"US:en"
    }
    args:["--incognito","--no-sandbox","--single-process",]
})

console.log(news)

Options

The options provided for configuring the behavior of the GNewsScraper has the following properties:

searchQuery (required)

Specifies the search term or keyword to find articles related to.

prettyUrl (optional)

Determines whether the scraper should follow redirects and retrieve the actual "pretty" URL of each article. Enabling this feature may impact the performance of the scraper due to additional HTTP requests.

without prettyUrl :

https://news.google.com/articles/CBMiZ2h0dHBzOi8vd3d3Lm5hc2RhcS5jb20vYXJ0aWNsZXMvZm9yZXgtZG9sbGFyLWZpcm1zLWFtaWQtc3VwcG9ydGl2ZS11LnMuLWRhdGEtc3dlZGlzaC1jcm93bi1hdXNzaWUtc2xpZGXSAQA?hl=en-US&gl=US&ceid=US%3Aen

with prettyUrl :

https://www.nasdaq.com/articles/forex-dollar-firms-amid-supportive-u.s.-data-swedish-crown-aussie-slide

cache (optional)

Indicates whether the scraper should cache the results to improve efficiency and reduce the number of requests to Google News. Using cache prevents blocking of IP by google news.

default:false

cacheTTL (optional)

Specifies the time-to-live (TTL) duration in seconds for the cached results. After the TTL expires, the scraper will fetch fresh results from Google News.

default:3600

timeframe (optional)

Filters the results to articles published within a specific timeframe before the request.

default:'7d'

The format of the timeframe is a string comprised of a number :

  • h = hours (eg: 1h)

  • d = days (eg: 1d , 7d)

  • y = years (eg: 1y)

queryParams (optional)

Additional query parameters to include in the URL when performing the search.

  • hl - represents the language (English-US)

  • gl - represents the country (US)

  • ceid - represents the country-specific identifier.

args (optional)

Additional arguements to included when launching puppeteer.

default:["--incognito",
"--no-sandbox",
"--single-process",
"--no-zygote",
"--disable-dev-shm-usage",
"--disable-setuid-sandbox"]

Available arguements

Check out the full list of puppeteer flags.

Output

The output is an array of JSON objects as mentioned below :

[
  {
    title: 'Nasdaq to transfer European power business to EEX bourse',
    articleUrl: 'https://www.reuters.com/business/energy/nasdaq-transfer-european-power-business-eex-bourse-2023-06-20/',
    sourceName: 'Reuters.com',
    imageUrl: 'https://lh3.googleusercontent.com/proxy/khH_YuxJvTa6rmavuuRIHG6JPu3_YD5b_jPjHZGn3t-5pfMORfSHJcuuTVCPdgRK8U_uD81JpOCFVe2YUpEGmhGNAEkm49VGoLgiKuDi2PZec-J4InGcSMtw4YIC5dwBcxngzUPv9MFK01fK3vl3ESvL8KF__Vs=s0-w100-h100-rw-dckaGU07gH',
    time: '19 hours ago'
  },
 ...
]

Upkeep

Please note that this is a web-scraper, which relies on DOM selectors, so any fundamental changes in the markup on the Google News site will probably break this tool. I'll try my best to keep it up-to-date, but changes to the markup on Google News will be silent and therefore difficult to keep track of. Feel free to submit an issue if the tool stops working.

Contribute

Feel free to submit a PR if you've fixed an open issue.

Issues

Please report bugs via the issue tracker.

gnews-scraper's People

Contributors

dstark5 avatar etrock avatar harikrishna-28 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

gnews-scraper's Issues

Scraping is not working

I cloned a repository and ran the tests and failed. Scraper doesn't work properly, doesn't return any response.

`
Gnews without pretty url
[
'--incognito',
'--no-sandbox',
'--single-process',
'--no-zygote',
'--disable-dev-shm-usage',
'--disable-setuid-sandbox'
]

pageUrl https://news.google.com/search?q=nasdaq&hl=en-US&gl=US&ceid=US:en
here
โœ” should return array (4436ms)
1) array should contain object
2) should have article url of "news.google.com"
`

I added some logs

Requesting additional optional parameter for puppeteer launch config

Description

The user should be able to set custom config while launching puppeteer.

Current behavior

There is no option available in Gnews that lets the user override the puppeteer launch arguments. The application has a set of default arguments not allowing the user to set custom arguments based on their requirements.

Proposed fix

An extra optional parameter in Gnews can be added that takes arg. It is an array of puppeteer launch configs that could be set by the user. Absence of which can result in the usage of default arguements.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.