Set of tasks for Loktra. Software engineering programming tasks.
This has been implemented successfully. Few lines are commented but can be uncommented for step by step visualization.
A module name URLparser has been provided that has a couple of functions for manipulating the uri. Run uri.py to test all the functions. As of now, the module has functions for splitting the uri into its respective components, get the scheme, domain and queries present. Still in development mode and has scope for more and better functionalities.
Implemented a web crawler in NodeJS for scraping the website www.shopping.com
Usage: node crawler.js [page_number] keyword
The web crawler performs 2 types of queries depending of the arguments given:
- If only the keyword is given, the crawler fetches the total number of results for the given keyword.
- If the page number and keyword are provided, the crawler fetches all the results for the given keyword in the specified page.