UNBREAKYou commented this keyword
Scrapling
A Python library and command-line tool that pulls content off websites, including dynamic and bot-protected pages, with an MCP server for AI agents.
- Python 3.10+
- Terminal
- Setup
- 10 min
- Level
- Intermediate
- Stars
- 86.0k
- License
- BSD-3-Clause
What it does
Scrapling fetches web pages and pulls out the parts you want. Its parser can find your elements again after a website changes its layout, and it has fetchers for plain requests, full browser automation and stealth mode.
You can use it from Python, or straight from the terminal without writing code. It also ships an MCP server (a connector that gives Claude a new tool) so AI agents can scrape through it, plus an Agent Skill that teaches coding agents the library.
Install it
- Make sure you have Python 3.10 or newer.
- Install Scrapling with the fetchers (the parts that actually download pages). The plain `pip install scrapling` only gives you the parser.
- Run `scrapling install` once to download the browsers it needs.
- Optional: install the MCP server extra with `pip install "scrapling[ai]"`, then follow the MCP server guide in the docs to connect it to Claude.
Copy this
pip install "scrapling[fetchers]" scrapling install
Use it
- Save a page as clean Markdown without writing code: `scrapling extract get 'https://example.com' content.md`. End the file name in .txt for plain text or .html for raw HTML.
- Add `--css-selector` to grab only part of a page.
- Run `scrapling shell` for an interactive scraping shell (needs the shell extra: `pip install "scrapling[shell]"`).
- In Claude Code, ask Claude to write a Python script with Scrapling for the site you want. The examples in the repo show the Fetcher, StealthyFetcher and Spider classes.
Try it
scrapling extract get 'https://example.com' content.md
Good to know
The README says the library is for educational and research purposes. You are responsible for following scraping and privacy laws, each site's terms of service and robots.txt.
A Docker image with all extras and browsers is available: `docker pull pyd4vinci/scrapling`.
Want the next one first?
One new guide a week, straight to your inbox.
