Pillar guide · Data collection
Web Scraping: Proxies, Tools & Success Rates in 2026
Everything that decides whether a crawl succeeds — the proxy tier each target demands, the tools worth running, measured anti-bot success rates, real cost per million pages, and the compliance lines not to cross.
Tools benchmarked
14
Requests per target
10,000
Regions tested
3
Test window
72 h
What web scraping actually costs you
Web scraping is the automated collection of publicly available web data into a structured dataset. The code is rarely the hard part. Almost every stalled scraping project we review has the same shape: a competent crawler pointed at a hostile target through the wrong exit IPs, burning bandwidth on retries that were never going to succeed.
Three variables decide the outcome — the proxy tier behind the request, whether the page truly needs a real browser, and how closely your fingerprint matches the client you claim to be. Get those right and a modest crawler clears targets that defeat expensive managed platforms. Get them wrong and no tool will save the run.
Proxy tiers for scraping — what each target needs
| Tier | Use when | Success rate | Cost | Added latency |
|---|---|---|---|---|
| Tier 1 — Datacenter | Static HTML, sitemaps, APIs, tolerant or internal targets | 97% on tolerant targets, 41% on protected ones | $0.30 – $2 per IP / month | 40 – 120 ms |
| Tier 2 — Rotating residential | Marketplaces, travel, classifieds, consumer sites | 89% median across protected targets | $1 – $8 per GB | 300 – 900 ms |
| Tier 3 — ISP / static residential | Logged-in dashboards, sticky sessions, account-bound crawls | 92% with sticky sessions over 30 min | $1.50 – $6 per IP / month | 80 – 200 ms |
| Tier 4 — Mobile 4G/5G | App-only endpoints, mobile web, hardest consumer targets | 94% but with heavy variance | $4 – $20 per GB | 400 – 1,400 ms |
| Tier 5 — Unlocker API | The stubborn tail your own stack cannot clear | 96 – 99%, billed only on success | $0.20 – $3 per 1,000 requests | 1 – 6 s |
Compare the networks behind each tier in the provider comparison or the proxy type guides.
The five-stage scraping pipeline
01
Discover
Sitemaps, search endpoints and internal APIs first. Most sites hand you a structured index that is cheaper and more stable than crawling the HTML tree.
02
Fetch
Plain HTTP by default. Escalate to a real browser only for pages whose data is injected client-side — a headless browser costs roughly 15× more per page.
03
Route
Attach the cheapest proxy tier that clears the domain, tracked per domain rather than globally. Promote a domain only when its measured block rate passes your threshold.
04
Parse
Selectors with fallbacks, schema validation on every record, and a quarantine queue for rows that fail validation instead of silently writing nulls.
05
Monitor
Track success rate, bytes per record, cost per 1,000 records and parse-failure rate per domain. Layout changes show up as parse failures long before they show up as empty dashboards.
Top-rated scraping tools
The six highest-scoring tools from our benchmark — crawling frameworks, headless browsers, unlocker APIs and AI extractors, each graded on ease, scale ceiling, anti-bot resilience, documentation and value.
See all 14 reviewed scraping tools · head-to-head comparisons
What one million pages costs
| Workload | Proxy tier | Bandwidth | Estimated proxy spend |
|---|---|---|---|
| 1M static HTML pages (60 KB avg) | Datacenter | ~60 GB | $120 – $250 |
| 1M marketplace pages (250 KB avg) | Rotating residential | ~250 GB | $450 – $1,900 |
| 1M JS-rendered pages (1.8 MB avg) | Residential + browser | ~1.8 TB | $2,600 – $9,000 |
| 1M requests through an unlocker | Unlocker API | n/a | $200 – $3,000 |
Size your own run with the bandwidth calculator.
Why crawls get blocked — and what to change
TLS & HTTP/2 fingerprinting
Cloudflare, Akamai and DataDome read your JA3/JA4 and HTTP/2 frame order before any JavaScript runs. Use a client whose fingerprint matches the browser you claim to be — a Chrome user agent over a default Python TLS stack is the single most common blocking cause we see.
Behavioural scoring
Perfectly even request intervals and a flat crawl order look nothing like human traffic. Jitter delays, randomise crawl order and cap per-IP request rates against the target's own limits.
Header consistency
Accept-Language, sec-ch-ua, platform hints and IP geography must agree. A German residential exit sending en-US with a macOS hint on a Linux TLS fingerprint fails consistency checks instantly.
Session lifetime
Rotate per request on anonymous listings; keep sticky sessions for anything behind a login or a multi-step flow. Mixing the two on the same domain triggers session-anomaly rules.
Scraping responsibly
Public data collection is broadly accepted in most jurisdictions, but the surrounding rules are not optional. Respect robots directives and published rate limits, never scrape behind an authentication wall you do not own, avoid collecting personal data without a lawful basis, and treat copyrighted content as licensed material rather than raw input. Throttle so your crawl never degrades the target's service.
Our full position — including what we will not publish or recommend — is in the compliance and scraping policy. This page is guidance, not legal advice; take counsel before running a commercial programme.
Web scraping FAQs
What are the best proxies for web scraping in 2026?+
Rotating residential proxies for consumer-facing sites, datacenter proxies for tolerant or internal targets, ISP proxies for logged-in sessions, and mobile proxies for app-only endpoints. Production stacks tier them: start cheap, escalate per domain only when the measured block rate justifies the extra cost.
How many proxies do I need for web scraping?+
With a rotating gateway you buy concurrency and bandwidth, not IP counts. Divide requests per hour by the target's per-IP rate limit to size concurrency, then add a 20–60% retry factor for blocks and timeouts.
Are datacenter proxies enough for scraping?+
Often, yes — and they cost a fraction as much with far lower latency. Always test datacenter first on each domain and promote to residential only when your block rate on that domain climbs past a few percent.
Do I need a headless browser to scrape?+
Only for pages whose data is injected client-side and is not available from an underlying JSON endpoint. Browsers cost roughly 15× more per page in bandwidth and compute, so check the network tab for an API before reaching for one.
What does web scraping cost at scale?+
Bandwidth dominates. One million static pages at 60 KB runs roughly $120–$250 on datacenter proxies; the same volume of JS-heavy marketplace pages on residential can pass $9,000. Measure bytes per record early — it is the number that decides your bill.
Is web scraping legal?+
Collecting publicly accessible data is broadly accepted in most jurisdictions, but terms of service, rate limits, authentication walls, copyright and data-protection law all still apply. Never scrape behind a login you do not own, never collect personal data without a lawful basis, and take legal advice for commercial programmes.
How do I avoid getting blocked while scraping?+
Match your TLS and header fingerprint to the browser you claim to be, jitter your request timing, respect per-IP rate limits, use sticky sessions for authenticated flows, and route only the blocked remainder through an unlocker API rather than upgrading your whole crawl.