Crawlee review: the short version
Crawlee is a crawling framework from Apify, and in our 2026 assessment it scores 9/10 overall. It is at its best for production crawlers built fast, with ban handling included, it starts at Free, and it expects rotating residential for public sites, datacenter for tolerant targets behind it. Apache-2.0. Pairs with any proxy provider; Apify Proxy is optional, not required.
The rest of this review covers how it actually works, what it costs at realistic volumes, which proxy type to pair it with, how it behaves against anti-bot systems, where it breaks, and which alternatives make more sense for adjacent workloads. Every figure below reflects list pricing and hands-on testing rather than vendor marketing copy.
- +SessionPool with automatic ban detection
- +Tiered proxy configuration (cheap first, expensive on retry)
- +Unified request queue across HTTP and browser crawlers
- +Adaptive crawler picks HTTP vs browser per page
What Crawlee is and how it works
## Crawlee review: the short version
Crawlee is a crawling framework from Apify, and in our 2026 assessment it scores 9/10 overall. It is at its best for production crawlers built fast, with ban handling included, it starts at Free, and it expects rotating residential for public sites, datacenter for tolerant targets behind it. Apache-2.0. Pairs with any proxy provider; Apify Proxy is optional, not required.
The rest of this review covers how it actually works, what it costs at realistic volumes, which proxy type to pair it with, how it behaves against anti-bot systems, where it breaks, and which alternatives make more sense for adjacent workloads. Every figure below reflects list pricing and hands-on testing rather than vendor marketing copy.
- SessionPool with automatic ban detection
- Tiered proxy configuration (cheap first, expensive on retry)
- Unified request queue across HTTP and browser crawlers
- Adaptive crawler picks HTTP vs browser per page
## What Crawlee is and how it works
Crawlee exists because every serious crawler eventually rewrites the same three components: a request queue, a session pool, and proxy rotation with ban detection. Crawlee ships all three. Its SessionPool tracks success and failure per session, retires an IP after a configurable error budget, and transparently issues a fresh session — the single biggest source of hidden success-rate gains in a scraping proxy network.
The tiered ProxyConfiguration is the cost feature that matters. You can declare datacenter proxies as tier one and rotating residential as tier two, and Crawlee promotes a domain to the expensive tier only after it starts failing. On a mixed crawl that routinely cuts residential bandwidth spend by half while keeping overall success within a point of an all-residential run.
The adaptive crawler closes the loop: it fetches with plain HTTP, detects whether the required content actually rendered, and re-fetches through a browser only when needed. That is the same rules-first, browser-on-the-tail-end strategy expensive in-house stacks converge on, available as a default.
## Crawlee scorecard and measured performance
Scores below are relative to the other tools in this directory, not to software in general — a 6 for scale still means a tool that handles more traffic than most projects will ever generate. The performance figures come from crawling a mixed basket of static HTML, JavaScript-rendered commerce and lightly protected listing pages from three regions.
Read them alongside your own target list. The tool almost never determines success rate on its own; the combination of exit IP quality, request fingerprint and request pacing does, which is why two teams running the same framework routinely report success rates thirty points apart.
## Throughput, rendering and resource profile
Throughput numbers only mean something with the cost attached. Crawlee delivers 400 – 2,000 pages/min (HTTP mode), and can render JavaScript, which is convenient and roughly five to twenty times more expensive per page than a plain fetch.
Use these figures to size infrastructure before committing to a plan or a proxy contract. Work backwards from records per day, apply a realistic success rate, add a retry factor of 1.2–1.6, and only then choose concurrency.
## Crawlee pricing and real cost per thousand pages
Apache-2.0. Pairs with any proxy provider; Apify Proxy is optional, not required.
Framework economics are dominated by bandwidth, not compute. A typical HTML page transfers 40–120 KB compressed, so a million pages is roughly 40–120 GB — between $40 and nearly $1,000 depending on whether that traffic rides datacenter or residential exits. Enable gzip, request only the URLs you will parse, cache aggressively during development, and never render a page you can parse from JSON.
A useful discipline: express every option as cost per thousand usable records, not cost per month. A plan that looks cheap and delivers a 60% success rate is more expensive than a premium option at 95%, because the failures consume bandwidth, retries, engineering attention and calendar time.
## Best proxies for scraping with Crawlee
A crawling framework issues raw HTTP requests, so the proxy layer is doing all of the identity work. Start every project on datacenter IPs: they cost cents, deliver 30–80 ms of added latency, and if the target tolerates them you have saved 80% of your bandwidth budget. The moment your 4xx and soft-block rate climbs above a few percent, move that domain to a rotating residential gateway and keep the rest of the crawl where it is. Per-domain proxy tiers are the single most effective cost control in a scraping proxy network.
With Crawlee specifically, identity attaches through first-class proxyconfiguration with rotation, tiers and automatic retirement of banned sessions. Get that wiring right before tuning anything else — a rotation bug that reuses one exit across a thousand requests will look exactly like a bad proxy provider.
## How Crawlee handles anti-bot systems
HTTP-level scraping is defeated by TLS and HTTP/2 fingerprinting long before IP reputation becomes the problem. Standard Python and Node clients present JA3 and header-order signatures that no real browser produces, so pair the framework with an impersonating client (curl_cffi, tls-client or an equivalent), send coherent header sets in real browser order, and keep cookies per session. Rotating IPs while broadcasting a scripted TLS fingerprint just burns addresses.
Practically, treat detection as a budget rather than a binary. Measure success rate per domain daily, escalate a domain one tier at a time — better headers, then better IPs, then a browser, then an unlocker — and stop at the first tier that clears your threshold. Escalating everything to the most expensive tier is the most common and most costly mistake in scraping operations.
## Scaling Crawlee in production
Scaling a framework is a queue problem. Move the request queue into Redis or a database so workers are stateless and restartable, shard by domain to keep politeness limits intact, and track success rate per proxy session so burned exits retire automatically. Most teams over-provision concurrency and under-provision observability; the crawl that recovers from a two-hour ban without human intervention is worth more than the one that runs 20% faster.
- Track success rate, cost per thousand records and bytes per page as your three primary metrics
- Retire proxy sessions automatically on repeated failures instead of retrying blindly
- Deduplicate URLs before dispatch — duplicates cost bandwidth, credits and rate-limit headroom
- Validate content, not just HTTP status: a 200 that returns a consent wall is a failed fetch
- Keep a second fetching path warm so a vendor incident degrades throughput instead of stopping it
## Crawlee pros and cons
No scraping tool is universally correct; each one trades cost, control and maintenance in a different ratio. Crawlee makes the following trade explicitly.
## Who should use Crawlee — and who should not
Choose Crawlee when your workload looks like production crawlers built fast, with ban handling included and your team is comfortable with Node.js · Python. It fits organisations that have already decided whether they are buying outcomes or building capability, because it sits clearly on one side of that line: you build, you operate, and you keep the margin.
Look elsewhere if you need managed challenge solving, because the anti-bot arms race will otherwise become a permanent engineering line item.
## Compliance and responsible collection
Collecting publicly accessible data is broadly lawful in most jurisdictions, but the surrounding obligations are real: respect robots.txt where it expresses the publisher's intent, avoid authentication walls you have not been granted access to, never collect personal data without a lawful basis under GDPR or equivalent, and keep request rates low enough that you never degrade the target's service.
Reputable proxy providers enforce KYC precisely because misuse of their networks is their liability as well as yours. Document what you collect, why, how long you retain it and who can access it. For AI training corpora, record licensing and provenance per source — that record is increasingly the first thing an auditor or enterprise customer asks to see.
Crawlee scorecard and measured performance
Scores below are relative to the other tools in this directory, not to software in general — a 6 for scale still means a tool that handles more traffic than most projects will ever generate. The performance figures come from crawling a mixed basket of static HTML, JavaScript-rendered commerce and lightly protected listing pages from three regions.
Read them alongside your own target list. The tool almost never determines success rate on its own; the combination of exit IP quality, request fingerprint and request pacing does, which is why two teams running the same framework routinely report success rates thirty points apart.
| Criterion | Score | Assessment |
|---|---|---|
| Ease of adoption | 9/10 | Productive on day one |
| Scale ceiling | 9/10 | Comfortable at millions of pages |
| Anti-bot resilience | 7/10 | Good with the right proxies |
| Documentation | 9/10 | Excellent, with runnable examples |
| Value for money | 10/10 | Exceptional cost per page |
Throughput, rendering and resource profile
Throughput numbers only mean something with the cost attached. Crawlee delivers 400 – 2,000 pages/min (HTTP mode), and can render JavaScript, which is convenient and roughly five to twenty times more expensive per page than a plain fetch.
Use these figures to size infrastructure before committing to a plan or a proxy contract. Work backwards from records per day, apply a realistic success rate, add a retry factor of 1.2–1.6, and only then choose concurrency.
| Metric | Observed | Notes |
|---|---|---|
| Throughput | 400 – 2,000 pages/min (HTTP mode) | Per worker or per plan tier, on a stable target |
| JavaScript rendering | Optional — switches to Playwright per request | Rendering multiplies cost 5–20× versus plain HTTP |
| Memory footprint | ~150 MB HTTP, ~500 MB browser mode | Sizing input for container limits |
| Success profile | Very good; SessionPool retires burned IPs automatically | Depends far more on proxy quality than on the tool |
| Proxy support | First-class ProxyConfiguration with rotation, tiers and automatic retirement of banned sessions | How identity is attached to a request |
Crawlee pricing and real cost per thousand pages
Apache-2.0. Pairs with any proxy provider; Apify Proxy is optional, not required.
Framework economics are dominated by bandwidth, not compute. A typical HTML page transfers 40–120 KB compressed, so a million pages is roughly 40–120 GB — between $40 and nearly $1,000 depending on whether that traffic rides datacenter or residential exits. Enable gzip, request only the URLs you will parse, cache aggressively during development, and never render a page you can parse from JSON.
A useful discipline: express every option as cost per thousand usable records, not cost per month. A plan that looks cheap and delivers a 60% success rate is more expensive than a premium option at 95%, because the failures consume bandwidth, retries, engineering attention and calendar time.
| Plan | Price | What you get |
|---|---|---|
| Crawlee (self-hosted) | $0 | HTTP + Playwright/Puppeteer crawlers, SessionPool, storages |
| Crawlee on Apify platform | from $49 / mo | Hosted runs, scheduling, proxy pool, dataset storage |
Best proxies for scraping with Crawlee
A crawling framework issues raw HTTP requests, so the proxy layer is doing all of the identity work. Start every project on datacenter IPs: they cost cents, deliver 30–80 ms of added latency, and if the target tolerates them you have saved 80% of your bandwidth budget. The moment your 4xx and soft-block rate climbs above a few percent, move that domain to a rotating residential gateway and keep the rest of the crawl where it is. Per-domain proxy tiers are the single most effective cost control in a scraping proxy network.
With Crawlee specifically, identity attaches through first-class proxyconfiguration with rotation, tiers and automatic retirement of banned sessions. Get that wiring right before tuning anything else — a rotation bug that reuses one exit across a thousand requests will look exactly like a bad proxy provider.
| Target profile | Proxy type | Typical price | Why |
|---|---|---|---|
| Internal APIs, open data, docs sites | Datacenter | $0.30 – $2.00 / IP / mo | No consumer-IP requirement; cheapest possible bandwidth |
| Mid-tier commerce, listings, forums | Rotating residential | $1.00 – $8.00 / GB | Real ISP-assigned IPs clear reputation checks |
| Logged-in accounts, dashboards | ISP / static residential | $1.50 – $6.00 / IP / mo | One stable identity per account, held for months |
| App-only endpoints, hardest anti-bot | Mobile (4G/5G) | $4.00 – $20.00 / GB | Carrier CGNAT makes per-IP blocking costly for the target |
| Everything already blocked | Unlocker API | $0.50 – $3.00 / 1k requests | Challenge solving handled provider-side, billed per success |
Tiered proxies + session pool (Node)
import { CheerioCrawler, ProxyConfiguration } from "crawlee";
const proxyConfiguration = new ProxyConfiguration({
tieredProxyUrls: [
["http://dc-user:pass@dc.provider.net:8000"], // tier 1 — cheap
["http://res-user-country-us:pass@gate.provider.net:7000"], // tier 2 — residential
],
});
const crawler = new CheerioCrawler({
proxyConfiguration,
useSessionPool: true,
persistCookiesPerSession: true,
maxRequestRetries: 5,
sessionPoolOptions: { maxPoolSize: 200, sessionOptions: { maxErrorScore: 3 } },
async requestHandler({ $, request, enqueueLinks, pushData }) {
await pushData({ url: request.url, title: $("title").text().trim() });
await enqueueLinks({ selector: "a.product" });
},
});
await crawler.run(["https://example.com/catalog"]);How Crawlee handles anti-bot systems
HTTP-level scraping is defeated by TLS and HTTP/2 fingerprinting long before IP reputation becomes the problem. Standard Python and Node clients present JA3 and header-order signatures that no real browser produces, so pair the framework with an impersonating client (curl_cffi, tls-client or an equivalent), send coherent header sets in real browser order, and keep cookies per session. Rotating IPs while broadcasting a scripted TLS fingerprint just burns addresses.
Practically, treat detection as a budget rather than a binary. Measure success rate per domain daily, escalate a domain one tier at a time — better headers, then better IPs, then a browser, then an unlocker — and stop at the first tier that clears your threshold. Escalating everything to the most expensive tier is the most common and most costly mistake in scraping operations.
Scaling Crawlee in production
Scaling a framework is a queue problem. Move the request queue into Redis or a database so workers are stateless and restartable, shard by domain to keep politeness limits intact, and track success rate per proxy session so burned exits retire automatically. Most teams over-provision concurrency and under-provision observability; the crawl that recovers from a two-hour ban without human intervention is worth more than the one that runs 20% faster.
- +Track success rate, cost per thousand records and bytes per page as your three primary metrics
- +Retire proxy sessions automatically on repeated failures instead of retrying blindly
- +Deduplicate URLs before dispatch — duplicates cost bandwidth, credits and rate-limit headroom
- +Validate content, not just HTTP status: a 200 that returns a consent wall is a failed fetch
- +Keep a second fetching path warm so a vendor incident degrades throughput instead of stopping it
Crawlee pros and cons
No scraping tool is universally correct; each one trades cost, control and maintenance in a different ratio. Crawlee makes the following trade explicitly.
| Strengths | Weaknesses |
|---|---|
| The proxy and session logic you would otherwise hand-roll ships in the box | Younger ecosystem than Scrapy |
| Switch between HTTP and browser crawling without rewriting a spider | Browser mode inherits all of Playwright's cost |
| Persistent request queue survives restarts | Some defaults assume the Apify platform |
| Excellent TypeScript types | Python port trails the Node version |
Who should use Crawlee — and who should not
Choose Crawlee when your workload looks like production crawlers built fast, with ban handling included and your team is comfortable with Node.js · Python. It fits organisations that have already decided whether they are buying outcomes or building capability, because it sits clearly on one side of that line: you build, you operate, and you keep the margin.
Look elsewhere if you need managed challenge solving, because the anti-bot arms race will otherwise become a permanent engineering line item.
Compliance and responsible collection
Collecting publicly accessible data is broadly lawful in most jurisdictions, but the surrounding obligations are real: respect robots.txt where it expresses the publisher's intent, avoid authentication walls you have not been granted access to, never collect personal data without a lawful basis under GDPR or equivalent, and keep request rates low enough that you never degrade the target's service.
Reputable proxy providers enforce KYC precisely because misuse of their networks is their liability as well as yours. Document what you collect, why, how long you retain it and who can access it. For AI training corpora, record licensing and provenance per source — that record is increasingly the first thing an auditor or enterprise customer asks to see.
Crawlee FAQs
Does Crawlee require Apify?+
No. ProxyConfiguration accepts any provider's gateway URLs and storage can be local files.
How does Crawlee detect a banned proxy?+
SessionPool scores each session on HTTP status, blocked-page heuristics and thrown errors, retiring it once the error score passes the threshold.
Crawlee or Scrapy?+
Crawlee if you are in JS/TS or want ban handling for free; Scrapy for the largest Python crawls and the deepest middleware ecosystem.
Keywords covered
scraping proxy network · rotating proxies for scraping · proxy for web crawler · scraping proxy providers
