Proxy Compliance & Scraping Policy

Using a proxy is lawful in most jurisdictions — proxies are ordinary network infrastructure used by enterprises for QA, ad verification, security research and market intelligence. Legality depends on what you do through the proxy: collecting public, non-personal data at a reasonable rate is generally defensible, while bypassing logins, ignoring an IP ban aimed at you, harvesting personal data without a lawful basis or overloading a target can be unlawful regardless of the proxy.

robots.txt is a voluntary standard (RFC 9309) rather than a statute in most jurisdictions, so it is rarely binding on its own. In practice it matters a great deal: courts and regulators treat a documented, deliberate disregard of a published crawl preference as evidence of bad faith, and under EU DSM Article 4 a machine-readable opt-out — which robots.txt AI tokens express — removes the text-and-data-mining exception. Rotating proxies do not change which robots.txt group applies to your crawler, because the group is selected by User-Agent, not by IP.

As a default, one request per second per host for a mid-size commercial site, one every three to five seconds for a small independent site, and up to five to ten per second only for large platforms that document equivalent API quotas. The budget is per target host, not per proxy IP: spreading 600 requests per minute across 600 residential IPs still delivers 600 requests per minute to the origin server.

Rotating IPs to distribute ordinary crawl load is normal network behaviour. Rotating specifically to defeat a block that was applied to you — especially after a cease-and-desist or an account suspension — is the fact pattern most likely to be characterised as circumventing an access control, and it materially strengthens a CFAA or unauthorised-access argument against you. If you are blocked, the compliant response is to slow down and make contact, not to re-enter from a different exit node.

Only for accounts you legitimately own and where the terms of service permit automated access, or where you have written authorisation from the site operator. Authenticated content is the clearest boundary between public and private data: accepting clickwrap terms makes them enforceable against you, and using someone else's credentials adds an unauthorised-access exposure on top of the contract breach.

Public availability is not a lawful basis under the GDPR. You need one — normally legitimate interests under Article 6(1)(f), supported by a documented balancing test — plus purpose limitation, data minimisation, a retention schedule, a route to satisfy access and erasure requests, and Article 14 transparency to the individuals whose data you collected indirectly. Special-category data such as health, political opinions, religion or biometrics requires a far higher bar and is best excluded from collection entirely.

Article 4 permits text and data mining of lawfully accessible works for any purpose, including commercial AI training, unless the rightsholder has expressly reserved the right in a machine-readable way. In practice that reservation is expressed through robots.txt directives targeting AI tokens such as GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot, through TDM Reservation Protocol headers, or through explicit terms. If a reservation exists and you mine anyway, the exception no longer covers you.

The compliance risk is comparable; the sourcing risk differs. Datacenter IPs come from hosting ASNs with clear provenance. Residential and mobile pools depend on how the peer devices were enrolled: consented SDK rewards and paid opt-ins are ethically sound, whereas silently bundled SDKs are not, and using such a pool means your traffic is leaving devices whose owners did not agree. Ask any residential provider for their consent model and their KYC process before you buy.

It is not legally mandatory in most jurisdictions, but it is the single cheapest risk reduction available. A descriptive token with a contact URL — for example MyCompanyBot/1.0 (+https://example.com/bot) — lets a site operator email you instead of escalating, and it makes your robots.txt compliance verifiable. Never spoof Googlebot or another verified crawler: major sites confirm those crawlers by reverse DNS and treat impersonation as hostile.

Price data is factual and generally not protected by copyright, which makes competitive price monitoring one of the more defensible use cases. The exposure sits elsewhere: the retailer's terms of service if you use an account, the EU database right if you extract a substantial part of a structured catalogue, and load-based claims if your crawl is aggressive. Keep volumes modest, store derived metrics rather than verbatim page copies, and refresh only the SKUs you actually need.

Treat 429 as an instruction, not an error. Honour the Retry-After header exactly when present; when it is absent, back off exponentially with jitter — 2s, 4s, 8s, 16s, capped around five minutes — and halve per-host concurrency after three consecutive 429s. Never retry immediately through a fresh proxy IP: that converts a polite throttle into an evasion pattern and is what triggers permanent blocks and abuse reports.

Anyone can file a claim, so the practical question is which theories are available. The common ones are breach of contract where terms were accepted, unauthorised access where a gate was bypassed, database right or copyright where substantial protected material was copied, trespass to chattels where load caused measurable harm, and data protection enforcement where personal data was involved. Public availability defeats some of these but not all — most defensibility comes from being slow, transparent, minimal and documented.

Special-category personal data (health, political opinions, religious beliefs, trade union membership, sexual orientation, biometrics and genetics), children's data, government-issued identifiers, credentials and session tokens, payment card data, and anything behind an access control you do not own. Providers also universally prohibit traffic aimed at critical infrastructure and government targets, and collection of illegal content.

It operates alongside the law and is enforced far faster. A provider AUP is a contract: activity it prohibits can get your account suspended and your balance forfeited within hours, even where the activity is lawful. Conversely, an AUP permitting something does not make it lawful in your jurisdiction. Check both, and get written confirmation from your provider that your specific use case is allowed before committing to volume.

Keep it only as long as the documented purpose requires, then delete it automatically. For price and SERP monitoring, most teams keep raw HTML for seven to thirty days and derived metrics indefinitely, since the aggregates carry no personal data. Where records contain personal data, set an explicit retention period tied to the purpose, implement scheduled deletion rather than manual cleanup, and be able to honour an erasure request across backups and derived stores.

No. Reputable providers retain connection metadata for abuse handling and disclose it under lawful process, payment and KYC records tie an account to a real entity, and target-side fingerprinting frequently links sessions across exit IPs. Treat a proxy as an attribution and geolocation tool, not as an anonymity guarantee, and build your programme so it withstands being identified rather than depending on not being.