Rank #03 · Daily · DevTools
Scrapling: One Python Framework From Single Request to Full Crawl
Fetch, parse, and crawl with one library, then hand the results to an LLM pipeline or let an agent drive it over MCP.
What it does
- Fetchers. A fast HTTP
Fetcherwith browser TLS impersonation.StealthyFetcherand Playwright-based browser fetchers handle JS-heavy or protected pages, with proxy rotation, session support, and optional Cloudflare Turnstile/Interstitial handling. - Parser. CSS/XPath selectors, BeautifulSoup-style finders, text/regex search, and adaptive element tracking that relocates elements after a site's layout changes. Pages convert to Markdown for LLM/RAG ingestion via
page.markdown(). - Spiders. A Scrapy-like
SpiderAPI with concurrency, multi-session routing, pause/resume (crawldir), AutoThrottle, robots.txt handling, and JSON/JSONL/CSV/XML exporters. Templates include CrawlSpider, SitemapSpider, and ShopifySpider. - Agent and CLI surface. An MCP server (
scrapling[ai]), an Agent Skill, an interactive shell, andscrapling extractone-liners.
Note on installation: pip install scrapling installs only the parser. Add "scrapling[fetchers]" and run scrapling install for browsers. Docker images ship with browsers included. Docs: scrapling.readthedocs.io.
Who it's for
Python developers who are tired of stitching together Requests, BeautifulSoup, Playwright, and Scrapy, and AI data-pipeline builders who need clean Markdown corpora or a scraping tool their agent can call. Follow site terms and robots.txt — the README frames the project for educational and research use.
Stars / language / license
96,589 stars · Python · BSD-3-Clause
Data verified: 2026-11-05 09:31 Asia/Shanghai (GitHub API get_repository + README get_file_contents)
Trending note: +2,005 stars today (Trending selection captured 09:26 Asia/Shanghai)
Category tags
DevTools · web-scraping · crawling · Python · MCP · data-pipelines · RAG
How it differs from similar projects
Compared with Scrapy, Scrapling adds stealth browser fetching and adaptive parsing in the same package. Compared with Playwright scripts, it adds spider orchestration and exporters. Compared with hosted "URL → Markdown" APIs (Firecrawl-style services), it's self-hosted, BSD-licensed, and a full crawler, not just a single-page converter. For AI pipelines, the pattern is: spider to collect pages, markdown() to clean them, MCP tools for ad-hoc agent lookups. The README's parser speed tables are project benchmarks that TidyHow hasn't reproduced.