Karim shoair · @D4Vinci ↗ Scrapling
Adaptive web scraping for Python: fetchers that bypass Cloudflare Turnstile, selectors that survive site redesigns, and a spider framework that scales from one request to a full crawl.
The roster
What it does
Scrapling is an adaptive web scraping framework for Python that gives your coding agent a full scraping stack: fast HTTP fetching that impersonates real browser fingerprints, full browser automation, and a stealth mode that clears Cloudflare Turnstile through automation alone, no captcha service or API keys.
Its parser survives redesigns. Save an element once and Scrapling relocates it by similarity after the layout changes, so your selectors stop rotting. Selection works by CSS, XPath, BeautifulSoup-style calls, text, or regex, at speeds in the same tier as Parsel and lxml.
When one page is not enough, the Scrapy-like spider framework runs concurrent crawls with per-domain throttling, pause and resume from checkpoints, and JSON or JSONL export. A CLI saves any page straight to Markdown, and an MCP server plugs the whole thing into AI tools.
When to use it
- You're picking a scraping stack for your coding agent
One free toolkit covers the whole job, from fast HTTP grabs to full browser automation, with a CLI and MCP server the agent can drive directly.
- Your scrapers break every time the site redesigns
Save an element once and Scrapling relocates it after the layout changes, so you stop rewriting selectors.
- You need the whole site, not one page
Scrapy-like spiders handle concurrent crawls with per-domain throttling, plus pause on Ctrl+C and resume from a checkpoint.
Common questions
Is Scrapling free to use?
Yes. Scrapling is free and open source under the BSD-3-Clause license, with no paid tier. You install it with pip and run everything on your own machine.
What do I need to run it?
Python 3.10 or newer. You install Scrapling with pip, then run scrapling install once to download the browsers the dynamic and stealth fetchers use. A Docker image covers machines without Python.
How is it different from BeautifulSoup or Scrapy?
Scrapling keeps a familiar Scrapy and BeautifulSoup style API but parses hundreds of times faster than BeautifulSoup, in the same speed tier as Parsel and lxml. A bundled guide covers migration from BeautifulSoup, and an integration runs the parser inside your existing Scrapy projects.
Is it legal to scrape websites with it?
Scrapling is published for educational and research use, and compliance stays with you: scraping and privacy laws still apply, and so do each site's terms. Spiders honor robots.txt automatically when you set robots_txt_obey=True.