firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Topic: ai-scraping
78 repositories
D4Vinci/Scrapling
🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
itsOwen/CyberScraper-2077
A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama
oxylabs/ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
0xMassi/webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
raznem/parsera
Lightweight library for scraping web-sites with LLMs
oxylabs/ai-scraper-py
AI Scraper is a powerful scraping tool and scrape agent built to automate data extraction with unmatched precision. Ideal for scalable AI scraping tasks across diverse web sources, this tool simplifies complex scraping operations into efficient, intelligent workflows.
firecrawl/firecrawl-app-examples
🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.
BrowserCash/teracrawl
High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative
WeebDataHoarder/go-away
[Mirror] Self-hosted abuse detection and rule enforcement against low-effort mass AI scraping and bots.
ArchiveBox/abx-dl
⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...
kaymen99/ai-web-scraper
AI web scraper built with Crawl4AI for extracting structured leads data from websites.
osintshifu/awesome-osint-repos
Continuously updated catalogue of open-source OSINT tools, skills, plugins, MCP servers, and agentic integrations.
kaymen99/google-maps-lead-generator
Extract Google Maps business leads and enrich contact details using AI & web scraping
telepat-io/otto
Automate web workflows on real browser tabs without hosting a browser farm.
spider-rs/web-crawling-guides
How to guides on web-crawling or scraping
spider-rs/spider-clients
Python, Javascript, and Rust libraries for the Spider Cloud API.
TurboNodes/Turbo
Fastest distributed residential proxy network.
ScrapeRouter/awesome-scraping
The definitive list of the latest libraries, tools, APIs and providers for web scraping. The only daily-updated collection of web scraping resources.
spidra-io/spidra-js
The official Node.js SDK for Spidra.
drisskhattabi6/AI-Scraper
AI Scraper : scrap and extract data from website in any format (CSV, JSON, HTML...) using Selenium or Crawl4ai, and using Ollama or Sambanova API, and using Streamlit for UI as chatbot
webscrape-ai/webscrape-ai
The web scraping API that turns any URL into clean data.
nathabonfim59/md-fetch
A CLI tool and REST API that converts web content to clean Markdown, bypassing anti-scraping measures using headless browsers. Perfect for AI/LLM applications
chrisabruce/scrapling-rs
Adaptive web scraping, built in Rust. A high-performance port of Python Scrapling.
luminati-io/openai-sdk-with-web-unlocker
Integrating OpenAI Agents SDK with Bright Data Web Unlocker, enabling AI agents to access, extract, and process structured data from protected web pages
Scrape-Technology/MarkUDown-Engine
High-performance web scraping engine that converts any web page into clean markdown --- with 3-layer fallback (Cheerio --> Playwright --> Abrasio) and AI-powered structured extraction
ScrapingBee/ScrapingBee
Web scraping API examples in Python, Node, Go, Java, PHP, Ruby, cURL and CLI. Proxies, JS rendering, AI extraction, 9 scrapers.
jacoobwang/beecrawl
The Web Data Infrastructure for AI