Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
Jul 2, 2025 - Python
#
scraping
Here are 6,790 public repositories matching this topic...
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
markdown crawler data scraper ai html-to-markdown web-crawler scraping webscraping rag llm ai-scraping
-
Updated
Jul 3, 2025 - TypeScript
AIHawk aims to easy job hunt process by automating the job application process. Utilizing artificial intelligence, it enables users to apply for multiple jobs in a tailored way.
python resume bot agent chrome scraper automation job scraping selenium jobs artificial-intelligence automate jobseeker gpt jobsearch human-resources chatgpt opeai application-resume
-
Updated
May 28, 2025 - Python
Python scraper based on AI
markdown crawler ai html-to-markdown web-crawler scraping web-scraping rag automated-scraper scraping-python web-crawlers llm ai-scraping
-
Updated
Jul 3, 2025 - Python
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
-
Updated
Jul 3, 2025 - TypeScript
🕵️♂️ Collect a dossier on a person by username from thousands of sites
python open-source osint parsing social-network scraping sherlock python3 cybersecurity identification username blueteam investigation redteam osint-framework username-checker socmint osint-python username-search namechecker
-
Updated
Jul 1, 2025 - Python
Pythonic HTML Parsing for Humans™
-
Updated
Apr 16, 2024 - Python
Custom Selenium Chromedriver | Zero-Config | Passes ALL bot mitigation systems (like Distil / Imperva/ Datadadome / CloudFlare IUAM)
testing chrome automation webdriver browser captcha scraping selenium navigator python3 cloudflare chromedriver anti-bot bot-detection cloudflare-bypass distil anti-detection
-
Updated
Jun 25, 2024 - Python
List of libraries, tools and APIs for web scraping and data processing.
crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python
-
Updated
Dec 27, 2024 - Makefile
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation
-
Updated
Jun 9, 2025 - Python
Declarative web scraping
go cli golang crawler chrome data-mining scraper library tool dsl scraping crawling query-language scraping-websites cdp
-
Updated
Jul 4, 2025 - Go
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
-
Updated
Jul 3, 2025 - Python
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
python crawler data automation ai scraping crawling web-scraper python3 web-scraping selectors xpath data-extraction stealth webscraping hacktoberfest crawling-python playwright web-scraping-python ai-scraping
-
Updated
Jun 29, 2025 - Python
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
nlp crawler text-mining html-to-markdown scraping news-aggregator text-extraction web-scraping rss-feed readability tei html2text news-crawler corpus-builder corpus-tools article-extractor text-cleaning rag text-preprocessing llm
-
Updated
May 30, 2025 - Python
Collection of useful data science topics along with articles, videos, and code
python data-science machine-learning natural-language-processing time-series scraping data-visualization artificial-intelligence data-analysis articles
-
Updated
Jun 16, 2025 - Jupyter Notebook
Improve this page
Add a description, image, and links to the scraping topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the scraping topic, visit your repo's landing page and select "manage topics."