serpapi / public-roadmap Goto Github PK

The Yandex search is returning the url and thumbnail for the inline_images without base domain.

https://serpapi.com/playground?engine=yandex&text=spider+man&no_cache=true

inspect

https://serpapi.com/searches/60756c99033e94e3ec23c2ff/inspect

Past Canny Link

X-Ray is highlighting element on the top and bottom carousels. It shouldn't highlight the bottom carousel element.
The position is also reset for the bottom carousel and X-Ray doesn't work properly
Inspect: https://serpapi.com/searches/61c112a796f5d71b1832e169/inspect

Past Canny Link

Inspect Link

This block shows up when you add tbs:local_avail:1 parameter.

Playground Link

Inspect Link

Screenshot:

Past Canny Link

Google Local Pack, and Google Local Results have problematic results where addresses are sometimes parsed as numbers etc.

Employing Machine Learning classification models to improve the parser would be a viable solution.

Related Blog Posts About the Process:

#1) How to scrape Google Local Results with Artificial Intelligence?
#2) Real World Example of Machine Learning on Rails
#3) AI Training Tips and Comparisons
#4) Machine Learning in Scraping with Rails
#5) Implementing ONNX models in Rails
#6) How ML Hybrid Parser Beats Traditional Parser
#7) How to Benchmark ML Implementations on Rails

The date doesn't seem to be extracted anymore on our top_stories block on Google regular.

Playground: https://serpapi.com/playground?q=Inflation
Inspect: https://serpapi.com/searches/6212dce866440aa54c47b955/inspect

Original report:

Ref: https://forum.serpapi.com/bugs/p/date-missing-from-topstories

Apple Reviews Scraper API's maximum depth limit itself to maximum of 10 pages. It needs to be increased to real maximum of the search.

Google shopping results is not parsing the sections below currently.

Past Canny Link

Inspect Link
Playground Link

Past Canny Link

Inspect: https://serpapi.com/searches/6081db17d5a5313c7cc3cfb9/inspect

Playground: https://serpapi.com/playground?q=Dymo+4XL+LabelWriter+Label+Printer&location=Australia&google_domain=google.com.au&gl=au&hl=en&tbm=shop

On Google Shopping results, if the result is also listed on Google Products, it will include an element that shows how many listings would be on the product page:

Past Canny Link

Past Canny Conversation:

Angelo: Emirhan Akdeniz, great! What's the ETA on that?

Emirhan Akdeniz: We are trying to separate Google Shopping as another engine. It would enable us to solve the issues much faster and effectively in the future. So far, every issue we heard from customers about Google Shopping, including this issue has been fixed in a pull request. We are reviewing last state of the update. My estimates on ETA is 1-2 weeks since it is a big review process. Thanks so much for your enthusiasm and patience.

Angelo: thank you!!

An user has noticed that the Google url has changed from chips to htchips. And the filters are not working anymore otherwise:

https://www.google.com/search?q=Barista&ibp=htl;jobs#htivrt=jobs&fpstate=tldetail&htichips=employment_type:CONTRACTOR&htischips=employment_type;CONTRACTOR&htidocid=rKKHXOEcrR4AAAAAAAAAAA%3D%3D

Justin tried to add this to the playground but it did not work because we don't support that parameter:

https://serpapi.com/playground?engine=google_jobs&q=Barista&htchips=employment_type%3BCONTRACTOR%26htidocid%3DrKKHXOEcrR4AAAAAAAAAAA%253D%253D&newPara=htchips

Original Canny issue: https://forum.serpapi.com/bugs/p/google-jobs-api-update-chips-parameter-to-htchips

Inspect Link

This is a different block that needs to be parsed under another key.

Playground Link
Inspect Link

Past Canny Link

Playground link: https://serpapi.com/playground?engine=naver&query=minecraft&where=web

Add a documentation page for answer box.

Add support for population answers. e.g. "population of China"

Playground: https://serpapi.com/playground?engine=baidu&q=%E4%B8%AD%E5%9B%BD%E4%BA%BA%E5%8F%A3

Google has a What to watch section in their search results and currently we aren't return data for it in our API.

Inspect: https://serpapi.com/searches/6225b8e61c450a597413aab4/inspect

Inspect Link

Support Apple Search Auto-completion Results Scraper API

Past Canny Link

More than a few users are looking to manually (not automatically) renew early their account.

If initiating this process is not foreseen to be creating such problems that we have had at November 5, 2021 billing issue then this will be very convenient for some of our users.

Please feel free to discuss but if no blocker then next for sure.

Google Shopping Results sometimes give "Searches Related To" part which could be helpful for customers.
Inspect Link

Past Canny Link

One of our users reached out to report that IM parameter of Baidu search is currently not working.

The Playground:

https://serpapi.com/playground?engine=baidu&q=Coffee&lm=7

The Working Version:

https://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=1&tn=baidu&wd=Coffee&lm=7&oq=Coffee&rsv_pq=c588bc6a00062f27&rsv_t=ad109P4046JsUeJFRnpaK4maauYaWCViFuox3SL6l2F%2B00U0BEuF5R5xZUw&rqlang=cn&rsv_enter=1&rsv_dl=tb&gpc=stf%3D1645438795%2C1646043595%7Cstftype%3D1&tfflag=1

Guidance from our user:

I got that url by...

• going to the url generated by your playground as you had set it up

• clicking on the filter tools (look for the little filter icon on the right under the search bar)

• the tools are dropdown menus and are revealed just as in google when you click on 'tools'

• the left-most dropdown is the one that controls the time.

• I set that to 'within 7 days', which is '一周内', the second option

• that successfully limited the results to the time specified and the first results page had the url above when I did this

By experimenting I found that the arguments that vary as I change the time limit are 'rsv_t' and 'gpc', but obviously the value of these parameters is encoded somehow.

next as we have received a couple of requests about this.

Welp, it seems like the only thing keeping SerpAPI from turning into a Super Saipan is Yelp!

But seriously, it would be nice to be able to scrape Yelp search results. Whether I like it or not, Yelp has the best combination of business names, reviews, and photos, and plenty of people trust what users on Yelp say. It’s a must for SerpAPI!

Walmart Product API returns empty results. Checked twice, those products return results but more than occasional hiccups preventing user completing their task.

Product ID: 913713226

Inspect: https://serpapi.com/searches/622117892236e2bcc473de66/inspect

{
  "search_metadata": {
    "id": "622117892236e2bcc473de66",
    "status": "Error",
    "json_endpoint": "https://serpapi.com/searches/d802734da53bc2d7/622117892236e2bcc473de66.json",
    "created_at": "2022-03-03 19:31:21 UTC",
    "processed_at": "2022-03-03 19:31:21 UTC",
    "walmart_product_url": "https://www.walmart.com/ip/913713226?selected=true",
    "total_time_taken": 450.98
  },
  "search_parameters": {
    "product_id": "913713226",
    "engine": "walmart_product",
    "device": "desktop"
  },
  "error": "We couldn't get valid results for this search. Please try again later. Set the no_cache parameter to true in order to force a retry."
}

The screenshot:

The Playground: https://serpapi.com/playground?engine=walmart_product&product_id=913713226

User also stated that they had problem with 994807896 but Playground seems fine:

https://serpapi.com/playground?engine=walmart_product&product_id=994807896

next.

The HTML in the Yandex Images engine doesn't work

https://serpapi.com/playground?engine=yandex_images&text=spider+man&no_cache=true
 

Inspect

One of our users reported that Google does not parse several elements from certain mobile results page. The differences here are that search device is mobile ad we are searching for videos of a darling of TikTok users: Charli D'amelio.

Missing elements in the results: (All numbered in the screenshot)

1- Video tags and their links
2- Youtube channel's link and a selection of videos
3- Short videos
4- People also search for (contents of this element is returned in related_searches key)
5- TikTok channel's link and a selection of videos
6- TikTik category link and a selection of videos
7- Broaden this search
8- Related searches (content of this element is not returned at all, returned content is from people also search for section)

The Playground:

https://serpapi.com/playground?q=charli+d%27amelio+videos&location=Austin%2C+Texas%2C+United+States&gl=us&hl=en&device=mobile

The Inspect:

https://serpapi.com/searches/6213ce7f49ecdb4ebc4202ed/inspect

Screenshot:

A part about general search information is being parsed as an organic result. It is causing an empty organic result. Also the information is not served in the result JSON.

The first product in SerpApi JSON got the wrong source into source field in Google Shopping API. CSS seems be leaking.

Playground: https://serpapi.com/playground?q=iphone+12+mini&location=Germany&google_domain=google.de&gl=de&hl=de&tbm=shop

Inspect: https://serpapi.com/searches/6216b3680ead41878b9fdd9d/inspect

Google Ad Results API is missing phone number extraction from the "top" "block_position" ad results.

For more, have a look at the playground link or json_endpoint link.

Add support for direct answers. e.g. "How high is mount Fuji?"

Playground: https://serpapi.com/playground?engine=baidu&q=%E5%AF%8C%E5%A3%AB%E5%B1%B1%E6%9C%89%E5%A4%9A%E9%AB%98

Support top searches in Baidu served unrelated to provided query.

Google Shopping API no results for "dog supplies" and limited results for other basic queries

Playground:
https://serpapi.com/playground?q=dog+supplies&location=Austin%2C+Texas%2C+United+States&gl=us&hl=en&tbm=shop

Inspect:
https://serpapi.com/searches/61d7332bc47d3ca7b1f3d6b2/inspect

The Playground: (This returns results to me)

https://serpapi.com/playground?q=Callaway+Big+Bertha+B21+Driver&uule=585069ddee19ad271e9c4c75&google_domain=google.co.uk&gl=uk&hl=en&tbm=shop

Inspect examples:

https://serpapi.com/searches/61d82dad5f220316258218a3/inspect
https://serpapi.com/searches/61d82da5aa6c95fba3174ddd/inspect
https://serpapi.com/searches/61d82edf8cd8782812cb365f/inspect

Screenshot:

Related Second Issue

Inspect: https://serpapi.com/searches/61d82facaef2646991fb90dc/inspect

Screenshot:

Another example:

https://serpapi.com/searches/61dd5070b7b1cc879f1b71ff/inspect

Another example:
https://serpapi.com/searches/6202cdc4d5a53137721bb4b8/inspect

Another example: https://serpapi.com/searches/62091327c47d3cbb63618531/inspect

Past Canny Link

Google Search API inline_video_carousels is missing first channel link

Playground: https://serpapi.com/playground?q=addison+rae&location=Austin%2C+Texas%2C+United+States&gl=us&hl=en

Inspect: https://serpapi.com/searches/6213c8b749ecdb4f6252bde7/inspect

Another one (but currently seems to be working):

SerpApi parser replaces the first link in the container with this Youtube link.

Ref: Google Search API inline_video_carousels Missing First Channel Link

Inspect Link

Add support for calculator answers. e.g. "2+2"

Playground: https://serpapi.com/playground?engine=baidu&q=2%2B2

Inspect Link

Support the searches people also search for related to your query.

This repository is taking over from our old forum (https://forum.serpapi.com/):

We'll be migrating past reports from https://forum.serpapi.com/ to this repository. Feel free to re-add your issue if we haven't already.

Add support for weather answers. e.g. "What's the weather like in Beijing?"

Playground: https://serpapi.com/playground?engine=baidu&q=%E5%8C%97%E4%BA%AC%E7%9A%84%E5%A4%A9%E6%B0%94%E6%80%8E%E4%B9%88%E6%A0%B7%EF%BC%9F

It would be useful to include the simg code necessary to do an "All Sizes" similar image search in the json results for a reverse image search. Right now, the html version of the reverse image search includes the link to the appropriate search, but it is not included in the JSON, which makes automating that extra step difficult.

`

Support related search queries provided by Baidu related to provided query.

One of our users reached out to report that Yahoo Shopping API does not work properly when sent certain characters.

The Playground Query: Great Value 2% Reduced Fat Milk, 128 Fl Oz - No Result

https://serpapi.com/playground?engine=yahoo_shopping&p=Great+Value+2%25+Reduced+Fat+Milk%2C+128+Fl+Oz

The Playground Query: Great Value 2 Reduced Fat Milk, 128 Fl Oz - No Result

https://serpapi.com/playground?engine=yahoo_shopping&p=Great+Value+2+Reduced+Fat+Milk%2C+128+Fl+Oz

The Playground Query: Great Value 2 Reduced Fat Milk 128 Fl Oz - Successful when removed comma and percentage

https://serpapi.com/playground?engine=yahoo_shopping&p=Great+Value+2+Reduced+Fat+Milk+128+Fl+Oz

The user submitted video:

Inspect: https://serpapi.com/searches/6225c5858b7ed2c74b43ccdd/inspect

next.

Google Maps Review API does not always sort by highest rated reviews first:

data_id=0x3b080dafbed183bf:0x5951f316ba13a37e

The Playground: https://serpapi.com/playground?engine=google_maps_reviews&data_id=0x3b080dafbed183bf%3A0x5951f316ba13a37e&hl=en&sort_by=ratingHigh&no_cache=true

The Inspect: https://serpapi.com/searches/6229bd6f37b24ae465097169/inspect

I have described it as not working properly because it outputs blank results when it does not and it tends to work for some places and not for others:

The default on the Playground works for example:

data_id=0x89c259a61c75684f:0x79d31adb123348d2

Playground: https://serpapi.com/playground?engine=google_maps_reviews&data_id=0x89c259a61c75684f%3A0x79d31adb123348d2&hl=en&sort_by=ratingHigh&no_cache=true

Inspect: https://serpapi.com/searches/6229c9c67690dcd0a6b5c112/inspect

Another random example data_id=0x89c258ff0f7a7793:0xbedc0a7158ebea16

Playground: https://serpapi.com/playground?engine=google_maps_reviews&data_id=0x89c258ff0f7a7793%3A0xbedc0a7158ebea16&hl=en&sort_by=ratingHigh

This is working.

next.

Separate and make another engine out of tbm=shop parameter for the ease of use, development, and reviewing.

In Google Shopping Results tag is repeated within extensions for some results.

Inspect Link

Past Canny Link

Some CSS is parsed with some results in google shopping.

Inspect Link

Playground Link

Reoccurence of bug: Search Inspect.

Past Canny Link

When using tbs parameter in the playground its value is lowercased. tbs value is case sensitive and should not be lowercased, otherwise, the parameter will have no effect.

Add support for finance answers. e.g. "Google stock"

Playground: https://serpapi.com/playground?engine=baidu&q=%E8%B0%B7%E6%AD%8C%E8%82%A1%E7%A5%A8

The Baidu engine organic result has a rich_snippet now it returns empty extensions for some results:

https://serpapi.com/playground?engine=baidu&q=elon+musk&ct=2&no_cache=true

Inspect