Topic: scraper Goto Github

Some thing interesting about scraper

👇 Here are 8260 public repositories matching this topic...

aapatre / automatic-udemy-course-enroller-get-paid-udemy-courses-for-free

scraper,Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

User: aapatre

python scraping selenium python3 scraper

adyzng / jd-autobuy

scraper,Python爬虫，京东自动登录，在线抢购商品

User: adyzng

python crawler jingdong scraper

ahmadibrahiim / website-downloader

scraper,💡 Download the complete source code of any website (including all assets). [ Javascripts, Stylesheets, Images ] using Node.js

User: ahmadibrahiim

Home Page: https://website-downloader.onrender.com

assets downloader offline-web-pages scraper

aliparlakci / bulk-downloader-for-reddit

scraper,Downloads and archives content from reddit

User: aliparlakci

Home Page: https://pypi.org/project/bdfr

reddit imgur gfycat scraper python downloader archive

alirezamika / autoscraper

scraper,A Smart, Automatic, Fast and Lightweight Web Scraper for Python

User: alirezamika

ai artificial-intelligence automation crawler machine-learning python scrape scraper scraping web-scraping webautomation webscraping

apify / crawlee

scraper,Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Organization: apify

Home Page: https://crawlee.dev

web-scraping web-crawling npm headless-chrome puppeteer automation apify scraping crawling crawler

avnsx / fansly-downloader

scraper,Easy to use fansly.com content downloading tool. Written in python, but ships as a standalone Executable App for Windows too. Enjoy your Fansly content offline anytime, anywhere in the highest possible content resolution! Fully customizable to download in bulk or single: photos, videos & audio from timeline, messages, collection & specific posts 👍

User: avnsx

Home Page: https://fansly.com/

fansly datascraping python fansly-scraper fansly-downloader fansly-download windows macos linux image-download video-download portable downloader scraper cross-platform gui reddit video open-source database

brucedone / awesome-crawler

scraper,A collection of awesome web crawler,spider in different languages

User: brucedone

web-crawler crawler web-scraper spider node-crawler scraper awesome

cheeriojs / cheerio

scraper,The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

Organization: cheeriojs

Home Page: https://cheerio.js.org

cheerio jquery htmlparser2 dom htmlparser selector scraper parser html hacktoberfest

cinemagoer / cinemagoer

scraper,Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people, characters and companies

Organization: cinemagoer

Home Page: https://cinemagoer.github.io/

imdb movies actors cinema movie-database python database sql cast internet-movie-database

codelucas / newspaper

scraper,newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

User: codelucas

Home Page: https://goo.gl/VX41yK

python news crawler crawling scraper news-aggregator

edoardottt / cariddi

scraper,Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

User: edoardottt

Home Page: https://edoardoottavianelli.it

bugbounty crawler crawling endpoint-discovery endpoints go golang hacktoberfest infosec osint penetration-testing pentesting recon reconnaissance redteam scraper secret-keys secrets-detection security security-tools

evil0ctal / douyin_tiktok_download_api

scraper,🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

User: evil0ctal

Home Page: https://douyin.wtf

aiohttp api asgi async asyncio crawler douyin douyin-scraper douyin-tiktok-api douyin-tiktok-download fastapi no-watermark online-parsing python pywebio scraper spider tiktok tiktok-scraper web-scraping

extractus / article-extractor

scraper,To extract main article from given URL with Node.js

Organization: extractus

Home Page: https://extractor-demos.pages.dev/article-extractor

nodejs article-parser readability article article-extractor crawler extract scraper

facundoolano / google-play-scraper

scraper,Node.js scraper to get data from Google Play

User: facundoolano

google-play scraper crawler api nodejs

felipecsl / wombat

scraper,Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

User: felipecsl

Home Page: https://felipecsl.github.io/wombat/

ruby scraper crawler dsl

fent / node-ytdl-core

scraper,YouTube video downloader in javascript.

User: fent

node scraper video-downloader youtube youtube-downloader

geziyor / geziyor

scraper,Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Organization: geziyor

go scraping scraper crawler spider

go-rod / rod

scraper,A Devtools driver for web automation and scraping

Organization: go-rod

Home Page: https://go-rod.github.io

cdp chrome-headless chrome-devtools chrome-devtools-protocol headless web-scraping automation scraper devtools devtools-protocol

gocolly / colly

scraper,Elegant Scraper and Crawler Framework for Golang

Organization: gocolly

Home Page: https://go-colly.org/

golang scraper framework crawler scraping crawling spider go

guyueyingmu / avbook

scraper,AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

User: guyueyingmu

adult adult-video avmoo crawler database guzzlehttp javbus javlibrary laravel magnet magnet-link scraper spider

holgerd77 / django-dynamic-scraper

scraper,Creating Scrapy scrapers via the Django admin interface

User: holgerd77

Home Page: http://django-dynamic-scraper.readthedocs.io

python django scraper scraping scrapy spider webscraping

huaying / instagram-crawler

scraper,Get Instagram posts/profile/hashtag data without using Instagram API

User: huaying

instagram instagram-crawler python webdriver instagram-liker autoliker auto likers instagram-scraper instagram-bot

huginn / huginn

scraper,Create agents that monitor and act on your behalf. Your agents are standing by!

Organization: huginn

automation notifications scraper webscraping feedgenerator rss agent monitoring feed twitter-streaming

iawia002 / lux

scraper,👾 Fast and simple video download library and CLI tool written in Go

User: iawia002

downloader go crawler scraper video bilibili youtube youku iqiyi tumblr

ionicabizau / scrape-it

scraper,🔮 A Node.js scraper for humans.

User: ionicabizau

Home Page: http://ionicabizau.net/blog/30-how-to-write-a-web-scraper-in-node-js

scraper node-scraper hacktoberfest

jae-jae / querylist

scraper,:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

User: jae-jae

Home Page: https://querylist.cc

querylist crawler spider scraper

javscraper / emby.plugins.javscraper

scraper,Emby/Jellyfin 的一个日本电影刮削器插件，可以从某些网站抓取影片信息。

User: javscraper

Home Page: https://javscraper.com

emby jav-scraper jav plugin scraper fanart-poster synology japanese adult metadata

joeyism / linkedin_scraper

scraper,A library that scrapes Linkedin for user data

User: joeyism

linkedin linkedin-scraper scraper profile driver linkedin-profile linkedin-url scrapes-linkedin chrome firefox

justanotherarchivist / snscrape

scraper,A social networking service scraper in Python

User: justanotherarchivist

python scraper social-network social-media

justfoolingaround / animdl

scraper,A highly efficient, fast, powerful and light-weight anime downloader and streamer for your favorite anime.

User: justfoolingaround

anime download python scraper

lorey / mlscraper

scraper,🤖 Scrape data from HTML websites automatically by just providing examples

User: lorey

Home Page: https://pypi.org/project/mlscraper/

scraping crawling html machine-learning extraction-engine scraper crawler crawler-python

lucasjinreal / weibo_terminater

scraper,Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

User: lucasjinreal

scraper weibo sina corpus chinese chatbot

madawei2699 / mygptreader

scraper,A community-driven way to read and chat with AI bots - powered by chatGPT.

User: madawei2699

Home Page: https://www.i365.tech/

chatgpt slack-bot prompt crawler embedding gpt-35-turbo scraper ai reader hot-news

meetdeveloper / freedictionaryapi

scraper,There was no free Dictionary API on the web when I wanted one for my friend, so I created one.

User: meetdeveloper

Home Page: https://dictionaryapi.dev/

dictonary google api scraper google-dictionary dictionary-api free-api

metafates / mangal

scraper,📖 The most advanced (yet simple) cli manga downloader in the entire universe! Lua scrapers, export formats, anilist integration, fancy TUI and more!

User: metafates

anime cli go manga manga-downloader scraper tui manga-reader golang anilist

montferret / ferret

scraper,Declarative web scraping

Organization: montferret

Home Page: https://www.montferret.dev/

golang query-language data-mining scraping scraping-websites dsl cdp crawling scraper crawler

naibowang / easyspider

scraper,A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

User: naibowang

Home Page: https://www.easyspider.net

batch-processing batch-script code-free crawler data-collection frontend gui html input-parameters layman parameters robotics rpa scraper spider visual visualization visualprogramming web www

niespodd / browser-fingerprinting

scraper,Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

User: niespodd

Home Page: https://niespodd.github.io/browser-fingerprinting/

bot detection chromium stealth puppeteer scraper webscraping web automation chromium-browser