Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
Jul 4, 2025 - Python
#
web-scraping
Here are 6,777 public repositories matching this topic...
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
notifications monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection changedetection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring
-
Updated
Jul 4, 2025 - Python
Python scraper based on AI
markdown crawler ai html-to-markdown web-crawler scraping web-scraping rag automated-scraper scraping-python web-crawlers llm ai-scraping
-
Updated
Jul 3, 2025 - Python
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
-
Updated
Jul 3, 2025 - TypeScript
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
-
Updated
Mar 23, 2025 - Python
🔥 No Code Web Data Extraction Platform • Open-Source Alternative To Octoparse, ParseHub 🔥
api scraper automation browser web-scraper self-hosted web-scraping data-extraction webscraping agents browser-automation no-code web-automation rpa robotic-process-automation web-agent playwright website-to-api no-code-web-scraper web-scraping-agent
-
Updated
Jul 3, 2025 - TypeScript
Python APIs for web automation, testing, and bypassing bot-detection.
python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python
-
Updated
Jul 4, 2025 - Python
Lighter web automation with Python
-
Updated
Apr 28, 2025 - Python
List of libraries, tools and APIs for web scraping and data processing.
crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python
-
Updated
Dec 27, 2024 - Makefile
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation
-
Updated
Jun 9, 2025 - Python
A Chrome DevTools Protocol driver for web automation and scraping.
testing go golang scraper automation web chrome-devtools headless devtools crawling web-scraping cdp chrome-headless rod chrome-devtools-protocol devtools-protocol gorod
-
Updated
Dec 7, 2024 - Go
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
-
Updated
Jul 3, 2025 - Python
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping
-
Updated
Jun 29, 2025 - Python
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
nlp crawler text-mining html-to-markdown scraping news-aggregator text-extraction web-scraping rss-feed readability tei html2text news-crawler corpus-builder corpus-tools article-extractor text-cleaning rag text-preprocessing llm
-
Updated
May 30, 2025 - Python
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
http curl https http-client web-scraping fingerprinting ja3 tls-fingerprint curl-impersonate ja3-fingerprint http2-fingerprint akamai-fingerprint
-
Updated
Jul 4, 2025 - Python
Official Firecrawl MCP Server - Adds powerful web scraping to Cursor, Claude and any other LLM clients.
web-crawler web-scraping data-collection batch-processing content-extraction search-api claude llm-tools firecrawl model-context-protocol mcp-server firecrawl-ai javascript-rendering
-
Updated
Jul 3, 2025 - JavaScript
Self-hosted webscraper.
python docker kubernetes opensource helm scraping webscraper web-scraper self-hosted web-scraping web-scrapers webscraping playwright
-
Updated
Jun 18, 2025 - TypeScript
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api php http client json framework curl xml proxy restful class http-client http-proxy api-client web-scraper requests web-scraping php-curl web-service php-curl-library
-
Updated
Jul 3, 2025 - PHP
The Python Code Tutorials
python python-tutorials machine-learning natural-language-processing computer-vision text-classification tutorials python3 web-scraping face-detection scapy network-analysis network-programming programming-tutorial ethical-hacking network-security socket-programming scapy-tutorials
-
Updated
May 30, 2025 - Jupyter Notebook
Improve this page
Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."