firecrawl/firecrawl
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Topic: ai-crawler
75 repositories
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
oxylabs/ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
vakra-dev/reader
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
BrowserCash/teracrawl
High-performance web crawler API optimized for LLMs. Turn any search or website into clean Markdown using remote browsers. Firecrawl alternative
puneetindersingh/open-seo-crawler
Free SEO crawler, website audit and AI crawler checker. Self-hosted and open source: crawls a site for technical SEO issues and checks that Googlebot, Bingbot and AI crawlers (GPTBot, PerplexityBot, OAI-SearchBot) get the same readable page. CMS-aware, XLSX export, free Screaming Frog alternative.
lennyerik/crawl4ai-proxy
A simple proxy server to integrate crawl4ai with OpenWebUI
makralabs/makra
Makra is a memory layer between the web and your AI agents. It serves real-time structured data from the open web at the cost of a vector search.
agentmarkup/agentmarkup
Build-time llms.txt, JSON-LD, markdown mirrors, AI crawler controls, and validation for Vite, Astro, Next.js, Nuxt, and any static site.
kr3t3n/smolagents-video-script-generator
A sophisticated system that uses multiple AI agents to research, create, and polish video scripts for social media platforms. The system employs specialized agents for research, script writing, polishing, and evaluation to ensure high-quality, engaging content.
Saqierma/miaowageo
免费开源的 GEO 检测工具 | Free open-source GEO (Generative Engine Optimization) checker — 检测网站能否被 ChatGPT、Claude、Perplexity、Gemini 抓取、理解与引用
spidra-io/spidra-js
The official Node.js SDK for Spidra.
webscrape-ai/webscrape-ai
The web scraping API that turns any URL into clean data.
motiv8-team/go-ua-parser
High-performance, zero-allocation HTTP User-Agent parser for Go — browser, OS, device, bot & AI crawler detection with Client Hints support
JerryZhi/AI-Crawler-Detector
Tool for Fast Detection of Website/Server AI Crawler Blocking Policies（Not robots.txt）
maango-io/ai-policy.json
Machine-readable AI permissions for websites. A consolidated spec at /.well-known/ai-policy.json for declaring how AI agents may train on, search, or use your content.
crawlcore/scp-protocol
A collection-based format for serving clean, structured web content to AI training systems and search engines through pre-generated collections.
Everlast-Consulting-GmbH/launchgrade-web
Stack-agnostic web template + 2026 standards (BFSG/GDPR/CWV/CSP/AI-crawler) as three Claude Code skills: setup → design → audit.
jacoobwang/beecrawl
The Web Data Infrastructure for AI
pixeldownward/ai-social-media-agent
🤖 Generate high-quality social media posts effortlessly with this AI agent that researches, drafts, critiques, and finalizes content for you.
iliasabk/geolint
ESLint for AI search — audit AI crawler access, llms.txt, structured data and citability for ChatGPT, Perplexity, Claude & co.
kr3t3n/documentation-crawler
A powerful tool that crawls documentation websites and generates a clean, well-formatted markdown document. Built with FastAPI and support for multiple LLM providers (DeepSeek and Groq).
sonhm3029/Auto-Crawl-Any-Page-with-Large-Language-Model
Auto Crawl Any Page with Large Language Model
breadrock1/news-rss
There is simple project to scrape and collect news using rss and llm API based on rust.
TryGeoSuite/ai-crawler-bots
See which AI crawlers can read your site — GPTBot, ClaudeBot, PerplexityBot & 20 more. Curated, operator-sourced bot list + a zero-dep CLI and GitHub Action to audit robots.txt, test reachability, read access logs, and gate it all in CI.
Muhammadfaizanjanjua109/ai-visibility
Make your Node.js app visible and citable by AI crawlers (GPTBot, ClaudeBot, PerplexityBot). Auto-generate robots.txt, llms.txt & JSON-LD schema for AI SEO/AEO/GEO, plus bot-detection middleware and crawler analytics.
maxaeo/maxaeo-ai-visibility-mcp
Local-first MCP server for AI visibility audits: GEO/AEO, AI SEO, llms.txt, schema, sitemap and AI crawler readiness.
sharozdawa/robotstxt-ai
Visual robots.txt manager for the AI era. Block or allow GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers with one click.
GGeronik/consent-txt
Define AI permissions once. Emit the rules today's AI web actually uses. One manifest → robots.txt, AIPREF headers, X-Robots-Tag, Google-Extended, TDMRep.
WebCarePro/ai-audit-docker
Self-hosted Docker container for AI & GEO Readiness Audit Tool (Next.js + Ubuntu 26.04 + Nginx)