thibauts / node-google-search-scraper Goto Github PK

customSolver not working

On creating a server and sending the image to the browser and later posting the result to be sent through the callback returns

/Users/../.../check1.js:77
  if(err) throw err;
          ^

Error: Captcha decoding failed

with customSolver as

app.post('/api/bears', function(req, res) {

            console.log(req.body.user.name);

            var id = null, err = null; 

            callback(err, id, req.body.user.name);

            res.end();
        });

Hello,

I'm searching google voor all indexed pages of a website, and put them in an array.
When it's done i wanna do something with it.

Is there a way to get a callback when the search is all done? At this moment the script just quits when the search is done.

Thank you.

I have tried the following piece of code

if(message.content.startsWith ("/google")) { var lastmsg = message var msgArray = message.content.split('/google ') var searchoptions = { query: msgArray[1], host: 'www.google.com.au', lang: 'en', limit: 1 }; var firstSearch = false; scraper.search(searchoptions, function(err, url) { //this is called for each result if(err) throw err; if(firstSearch == false) { console.log(url) message.channel.sendMessage(url).catch(console.error); firstSearch = true; } });
As you can see in that code i made a creative work-around.

But I shouldn't have had to do that, the problem is that even though i set limit to 1 it still didn't change it to 1.

I even thought maybe it's adding a zero on the end? making it 10 anyway? So i changed it to 01 but that didn't work either.

Is there something I'm doing wrong?

P.S.
Ignore the if statement about message.content and anything else about messages. That's there because i am using this inside of a discord bot.

Hi,

I am getting following issue :

{ Error: getaddrinfo ENOTFOUND www.google.comundefined www.google.comundefined:443
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:57:26)
  errno: 'ENOTFOUND',
  code: 'ENOTFOUND',
  syscall: 'getaddrinfo',
  hostname: 'www.google.comundefined',
  host: 'www.google.comundefined',
  port: 443 }
/Users/pankaj/work/genderapp/index.js:27

And here is my code:

const csvFilePath = './names_score_alltime.csv';
const csv = require('csvtojson');
const scraper = require('google-search-scraper');
const DeathByCaptcha = require('deathbycaptcha');

const dbc = new DeathByCaptcha('pankaj2k9', '...mypassord....');
let names = [];
csv()
    .fromFile(csvFilePath)
    .then((jsonObj) => {
        jsonObj.forEach((singleScore) => {
            names.push(singleScore.first);
        })
    })



let options = {
    query: 'information theory',
    age: 'y',
    solver: dbc
};

let urls = [];
scraper.search(options, (err, url) => {
    console.log(err);
    if (err) throw err;
    urls.push(url);
    Promise.all(urls).then((values) => {
        console.log(values);
    });
});

@

I'm trying to limit to 1 or 3 results,
but no matter what limit under 10 - it still gives back 10.

Script isn't throwing any error nor results

How would you use multiple keywords? Would it just be an array?

When sending certain queries the following exception is thrown:

TypeError: Parameter "url" must be a string, not undefined
    at Url.parse (url.js:88:11)
    at Object.urlParse [as parse] (url.js:82:5)
    at .<anonymous> ({removed}\node_modules\google-search-scraper\index.js:102:27)
    at exports.each ({removed}\node_modules\cheerio\lib\api\traversing.js:226:24)
    at extractResults ({removed}\node_modules\google-search-scraper\index.js:95:21)
    at onPage ({removed}\node_modules\google-search-scraper\index.js:32:26)
    at Request._callback ({removed}\node_modules\google-search-scraper\index.js:86:9)
    at Request.self.callback ({removed}\node_modules\request\request.js:122:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)

Example query:
{ query: 'ca', limit: 1 }

it'd been real nice if it returned more than just the url, like title/description/date. is that even possible?

url.js:103
throw new TypeError('Parameter "url" must be a string, not ' + typeof url);
^

TypeError: Parameter "url" must be a string, not undefined
at Url.parse (url.js:103:11)
at Object.urlParse [as parse] (url.js:97:13)
at module.exports. (/home/wallace/development/bikeAppointment/node_modules/google-search-scraper/index.js:101:27)
at module.exports.exports.each (/home/wallace/development/bikeAppointment/node_modules/cheerio/lib/api/traversing.js:226:24)
at extractResults (/home/wallace/development/bikeAppointment/node_modules/google-search-scraper/index.js:95:21)
at onPage (/home/wallace/development/bikeAppointment/node_modules/google-search-scraper/index.js:32:26)
at Request._callback (/home/wallace/development/bikeAppointment/node_modules/google-search-scraper/index.js:86:9)
at Request.self.callback (/home/wallace/development/bikeAppointment/node_modules/google-search-scraper/node_modules/request/request.js:122:22)
at emitTwo (events.js:126:13)
at Request.emit (events.js:214:7)

I am getting this error while trying to execute on node

    if(error) throw error;
                    ^
   Error: getaddrinfo ENOTFOUND www.google.com
   at errnoException (dns.js:44:10)
   at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:94:26)

here is my piece of code

var scraper = require('./node_modules/google-search-scraper');
var options = {
query: 'Nodejs',
limit: 10
}
scraper.search(options, function(error,url){
if(error) throw error;
console.log(url);
});

I was making my own with an amazing proxy feature and then I saw this and just gave up but I am wondering if this has a way to add a proxy. I probably should just look at the source...

I know it's a bit of a newbie question, but I really couldn't wrap my head around it.

The callback is called for each response.

To answer my own question,

const searchGoogle = (query: string, limit: number = 10) => {
  return new Promise((resolve, reject) => {
    const results = [];

    GoogleSearchScraper.search({
      host: 'www.google.com',
      lang: 'en',
      limit,
      query: query + ' site:imdb.com/title/'
    }, (error, result) => {
      if (error) {
        reject(error);
      } else {
        results.push(result);

        if (limit === results.length) {
          resolve(results);
        }
      }
    });
  });
};

However, this will hang if there are no results for the query.

Am I overlooking something?

Umlaut characters (e.g. å ä ö) show up as question marks: �.