firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Topic: scraping
8,766 repositories
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
soxoj/maigret
🕵️♂️ Collect a dossier on a person by username from 6K websites
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
gocolly/colly
Elegant Scraper and Crawler Framework for Golang
psf/requests-html
Pythonic HTML Parsing for Humans™
ultrafunkamsterdam/undetected-chromedriver
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
daijro/camoufox
🦊 Anti-detect browser
code4craft/webmagic
A scalable web crawler framework for Java.
jo-inc/camofox-browser
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in Puppeteer/Playwright replacement.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
alirezamika/autoscraper
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
tabulapdf/tabula
Tabula is a tool for liberating data tables trapped inside PDF files
autoscrape-labs/pydoll
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
proxifly/free-proxy-list
🚀 Free HTTP, SOCKS4, & SOCKS5 proxy list * Updated every 5 minutes * and rotating proxy API (100+ countries)
yujiosaka/headless-chrome-crawler
Distributed crawler powered by Headless Chrome
jaypyles/Scraperr
Self-hosted webscraper.
bjesus/pipet
Swiss-army tool for scraping and extracting data from online assets, made for hackers
d60/twikit
Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot
sparklemotion/mechanize
Mechanize is a ruby library that makes automated web interaction easy.
CodeCutTech/Data-science
Collection of useful data science topics along with articles, videos, and code
snooppr/snoop
Snoop — инструмент разведки на основе открытых данных (OSINT world)
fake-useragent/fake-useragent
Up-to-date simple useragent faker with real world database
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
kevinzg/facebook-scraper
Scrape Facebook public pages without an API key
aapatre/Automatic-Udemy-Course-Enroller-GET-PAID-UDEMY-COURSES-for-FREE
Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!