firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Topic: web-scraper
3,050 repositories
Panniantong/Agent-Reach
Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
lexmount/moli
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
getmaxun/maxun
Turn any website into a structured API. Extract, automate, search and monitor the web.
BruceDone/awesome-crawler
A collection of awesome web crawler,spider in different languages
gosom/google-maps-scraper
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
jaypyles/Scraperr
Self-hosted webscraper.
arpit-omprakash/100ProjectsOfCode
A list of practical knowledge-building projects.
oxylabs/google-ai-mode-scraper
Scrape Google AI Mode responses without blocks on a large scale.
php-curl-class/php-curl-class
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
itsOwen/CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
oxylabs/how-to-scrape-amazon-product-data
The process of extracting product data from Amazon using Python, including titles, ratings, prices, images, and descriptions.
lncrawl/lightnovel-crawler
Generate and download e-books from online sources.
anaskhan96/soup
Web Scraper in Go, similar to BeautifulSoup
oxylabs/how-to-scrape-google-scholar
A guide for extracting titles, authors, and citations from Google Scholar using Python and Oxylabs SERP Scraper API.
oxylabs/how-to-scrape-amazon-prices
A code for extracting best-selling items, search results, and currently available deals from Amazon using Python and Oxylabs E-Commerce Scraper API.
gildas-lormeau/single-file-cli
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
tholian-network/stealth
:rocket: Stealth - Secure, Peer-to-Peer, Private and Automateable Web Browser/Scraper/Proxy
fastcrw/crw
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
juancarlospaco/faster-than-requests
Faster requests on Python 3
oxylabs/ai-scraper-py
AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched precision. Ideal for scalable AI scraping tasks across diverse web sources, this tool simplifies complex scraping operations into efficient, intelligent workflows.
0x676e67/wreq
An ergonomic, privacy-aware Rust HTTP Client
je-suis-tm/web-scraping
Detailed web scraping tutorials for dummies with financial data crawlers on Reddit WallStreetBets, CME (both options and futures), US Treasury, CFTC, LME, MacroTrends, SHFE and alternative data crawlers on Tomtom, BBC, Wall Street Journal, Al Jazeera, Reuters, Financial Times, Bloomberg, CNN, Fortune, The Economist
Oshan96/monkey-dl
Bulk download your favourite anime episodes from your favourite anime websites
postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
dondai44423/donsetch
Web fetch, search, and crawl for AI agents. Built from scratch in Rust. No keys, no accounts. AGPL v3.
cassidoo/scrapers
A list of scrapers from around the web.
lumpinif/deepcrawl
100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.