ammanchhetri / web-scraping

Basic Demonstration of Web Scraping...using BeautifulSoup

Web-Scraping

Basic Demonstration of Web Scraping

Web-Scraping is a technique of extracting Data you want from the different Websites. The information is collected and then exported into a format that is more useful for the user, be it a spreadsheet or an API.

Since web scraping requires a lot of requests made to a server from an IP address, the server may detect too many requests and may block the IP address to stop further scraping. To avoid blocking, proxies are used and scraping will continue working as the IP address is changed and won’t cause any issues. It also helps in hiding the machine’s IP address as it creates anonymity.

Suppose we want to do the price analysis of certain item from different websites....for a certain period...then we can use Web-Scraping technique....to get the data from different websites in certain interval of time..

In this I have use proxy from ZenRows , for obtaining IP's that will rotate on each request to the url.. They provide free 1k credits worth of IP Address which we can use for first 15 days after signup.

Refer

• https://app.zenrows.com/builder
• https://www.zenrows.com/faq#getting-started

Documentation for BeautifulSoup

• https://beautiful-soup-4.readthedocs.io/en/latest/