D4Vinci/Scrapling

What it is

Scrapling is an adaptive web scraping framework that covers everything from a single request to a full-scale crawl. Its parser can relocate elements after a website changes, its fetchers handle anti-bot systems like Cloudflare Turnstile, and its spider framework supports concurrent crawls with pause/resume and proxy rotation. It also ships an MCP server that lets AI chatbots and agents such as Claude and Cursor scrape through it.

Who it's for

• Web scrapers who want scrapes that survive website design changes
• Developers building concurrent, multi-session crawls in Python
• Users of AI agents (Claude, Cursor, etc.) who want scraping tools exposed through MCP
• Teams building RAG pipelines who need pages or whole sites converted to Markdown
• Scrapy users who want to parse responses with Scrapling's parser

Requirements

Examples

from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
StealthyFetcher.adaptive = True
p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)  # Fetch website under the radar!
products = p.css('.product', auto_save=True)                                        # Scrape data that survives website design changes!
products = p.css('.product', adaptive=True)                                         # Later, if the website structure changes, pass `adaptive=True` to find them!

from scrapling.spiders import Spider, Response

class MySpider(Spider):
  name = "demo"
  start_urls = ["https://example.com/"]

  async def parse(self, response: Response):
      for item in response.css('.product'):
          yield {"title": item.css('h2::text').get()}

MySpider().start()

page.markdown()

Pros & cons

Images