5l1v3r1 / mediumsearchspider

Basic Implenetation of Scrapy framework

mediumSearchSpider

Basic Implenetation of Scrapy framework

Step 1: Install Python3 and pip3. I would recommend installing the Anaconda distribution as it contains most of the required modules we would need. https://www.anaconda.com/

Step 2: Go through the documentation and make sure that you are familiar with setting up environments in Conda

Step 3: Install scrapy

            pip install scrapy

Step 4: Clone this repository

Step 5: CD into the cloned directory

Step 6: You can run the spider by the follwing command

            scrapy crawl stories -a searchString="###search term###" -o outputfilename