How to Scrape a Website Without Getting Blocked (2026)

My short answer to scraping a website without getting blocked: respect robots.txt and the site's terms, throttle requests to a human-plausible rate with exponential backoff, send complete and realistic browser headers, keep cookies consistent within sessions, and distribute legitimate high-volume traffic across rotating residential IPs while monitoring status codes for early block signals.

Getting blocked is not bad luck. It is a site's anti-bot stack correctly identifying traffic that does not look like a person browsing — or traffic it has explicitly asked to go away. This guide covers the engineering practices that keep legitimate scraping projects (price monitoring, SEO tracking, ad verification, AI training data, market research) running reliably, and it is explicit about the lines we do not help anyone cross. For the full picture of how scraping works end to end, start with our web scraping guide.

Start with the rules: robots.txt and terms of service

Before optimizing anything technical, establish whether you should be scraping the target at all.

1. Fetch and parse robots.txt. The file at the site root declares which paths are off-limits to crawlers and often includes a Crawl-delay directive. Honoring it is the single clearest signal that your project operates in good faith, and most mature scraping frameworks can enforce it automatically.
2. Read the terms of service. Some sites prohibit automated access outright; others restrict specific uses. Whether ToS clauses are enforceable varies by jurisdiction and case — our overview of whether web scraping is legal covers the major rulings — but knowing what the site says is a baseline obligation, not an optional extra.
3. Prefer official channels. If the site offers an API, a data feed, or a licensing program that covers your use case, use it. It is more stable than scraping and removes the blocking problem entirely.
4. Scope your collection. Take only the fields you need, only from public pages, and exclude anything that identifies individuals. Under GDPR and similar regimes, personal data carries obligations that most scraping projects are not equipped to meet.

Our compliance stance, stated plainly: ProxyFacts does not provide guidance for bypassing paywalls or login walls, defeating CAPTCHAs at scale via solver farms, scraping personal data, or automating purchases (sneaker bots, ticket scalping, account creation). Anti-blocking engineering, as covered here, is about making legitimate access reliable — not about gaining access you were never permitted to have.

Understand why sites block scrapers

Anti-bot systems look for patterns no human produces. Knowing the detection layers tells you what to fix:

Two implications follow. First, IP rotation alone fixes only one layer — a scraper with a rotating IP but a Python-default User-Agent still gets caught at layer two. Second, the layers compound: consistent headers with inconsistent sessions look more suspicious than either flaw alone. Effective anti-blocking work means making every layer tell the same, truthful story: a reasonable client making a reasonable number of requests.

Rate limiting and backoff: the highest-leverage fix

The fastest way to get blocked is to send requests as fast as your connection allows. This is also where I would start any block diagnosis, because the fix costs almost nothing:

• Set a conservative baseline. One request every few seconds per domain is a sane starting point for most sites. If robots.txt declares a Crawl-delay, treat it as a floor, not a suggestion.
• Add jitter. Randomize the delay between requests (for example, a base delay multiplied by a random factor between 0.5 and 1.5). Perfectly regular intervals are a machine signature.
• Implement exponential backoff. On HTTP 429 or 503, wait, then retry with a doubling delay: 1 second, 2, 4, 8, and so on, capped at a sensible maximum. If the response includes a Retry-After header, obey it exactly.
• Budget concurrency per domain, not globally. Twenty concurrent workers are fine spread across twenty sites; twenty workers on one site is a burst that looks like an attack.
• Schedule around peak hours. Crawling a retailer during its traffic peak competes with real customers for server resources. Off-peak crawling is both politer and less likely to trip load-based defenses.

A scraper that respects these rules often needs no further anti-blocking work at all for small and medium jobs. Everything below matters mainly as volume grows.

Send realistic, consistent headers

Default HTTP client headers advertise exactly what you are. Fix the obvious tells:

1. Set a real browser User-Agent — a current Chrome or Firefox string, not a library default and not a browser version that is years out of date.
2. Send the full header set that browser would send: Accept, Accept-Language, Accept-Encoding, and the Sec-CH-UA client-hint headers that modern Chrome includes. A Chrome User-Agent with no client hints is a contradiction fingerprinting systems catch.
3. Keep headers internally consistent. Accept-Language should match the geography your IP claims. A German residential IP paired with a lone en-US language header is a mismatch.
4. Set Referer plausibly. Pages deep in a site are normally reached from somewhere, not summoned from nowhere.
5. Rotate User-Agents sparingly and correctly. If you rotate, rotate the entire consistent header bundle together — never mix one browser's User-Agent with another's client hints.

If the target renders content with JavaScript and challenges non-browser clients, header work alone will not suffice; you will need a real headless browser, and at that point a managed scraping API often makes more sense than building the stack yourself — see our comparison of scraping APIs versus raw proxies for where the trade-off tips. If you only need blocked pages fetched, not parsed, a dedicated unblocker covers that middle ground — our web unblocker comparison breaks down the major products and their billing models.

Manage sessions and cookies deliberately

Sites use cookies to track a visitor's continuity. Scrapers get flagged when that continuity breaks:

• Accept and persist cookies within a logical session. A client that receives a session cookie and never sends it back looks broken or hostile.
• Keep the IP and the cookie jar paired. A session cookie that arrives from three countries in ten minutes is an impossible traveler. If you rotate IPs, rotate cookie jars with them.
• Match session type to the job. Stateless tasks — fetching thousands of independent product pages — suit per-request IP rotation with no persistent cookies. Stateful flows — paginating a search result set, stepping through a multi-page listing — need a sticky session where IP and cookies stay fixed for the duration.
• Expire sessions like a human would. Nobody browses one site continuously for 72 hours. Retire session identities after a plausible lifespan and start fresh.

Major residential proxy providers expose session control for exactly this reason. According to their product pages (fetched July 2026): Oxylabs offers rotating plus sticky sessions up to 24 hours; Decodo (formerly Smartproxy) offers per-request rotation and sticky sessions from minutes up to days; IPRoyal supports sticky intervals up to 7 days; and Bright Data offers both sticky and rotating modes. Which window you need depends entirely on how long your longest stateful flow runs — I would measure that flow before choosing a provider, not after.

Rotating residential IPs for legitimate volume

When a project legitimately needs scale — say, monitoring 100,000 product pages daily or checking search rankings from 20 countries — a single IP cannot carry the load without exceeding any reasonable per-IP rate. Distributing requests across a pool of residential IPs keeps the per-IP request rate human-plausible while the aggregate job completes on time. Residential IPs also carry normal consumer reputation, unlike datacenter ranges that many sites block wholesale — the fundamentals are covered in our explainer on what a residential proxy is.

Oxylabs, Bright Data, Decodo and IPRoyal publish the following relevant capabilities (all figures from their product and policy pages, fetched July 2026; pool sizes are vendor claims, not independently verified):

Note the last row. Every reputable provider vets customers because their business depends on keeping abuse off the network. Bright Data restricts residential network access to verified companies that pass a human-reviewed KYC process, per its KYC FAQ. Oxylabs requires a KYC form from every customer at signup with risk-based escalation, per its KYC and safety policy. Decodo runs automated fraud checks and KYC on every registration and actively blocks high-risk targets such as banking and ticketing, per its security and compliance page. IPRoyal runs KYC through the third-party provider iDenfy, per its KYC policy. If your use case cannot survive a vendor's compliance review, the problem is the use case, not the vendor.

Engineer’s take (Hinata): The pool-size row is the one I would weight least; past a few million IPs it stops predicting your block rate. Compare the sticky-session window against your longest stateful flow first, then the effective per-GB cost at your realistic monthly volume rather than the headline entry price. Residential traffic is metered per gigabyte, so the cheapest optimization is scope — every page you skip and every image you do not fetch is money back at these rates. And budget calendar time for the KYC review: the vetting that keeps abusers off these networks also gates you until you pass it.

Practical rotation guidance for legitimate volume:

• Size the pool to the rate, not the ego. What matters is requests per IP per hour staying human-plausible on your specific target, not the headline pool number. Proxyway's Proxy Market Research 2026 (data collected March–April 2026) puts the median advertised residential pool at 54M IPs — every major provider has more addresses than any legitimate job needs.
• Use geo targeting purposefully. If you monitor German prices, request German IPs and send German-consistent headers. Geo targeting is for seeing what real local users see, not for evading region-level bans.
• Keep rotation and session strategy aligned, as described in the sessions section above.

For provider selection criteria beyond blocking — pricing structures, trials, support — see our best residential proxies roundup.

Why free proxies make blocking worse, not better

The tempting shortcut fails on the evidence. A 30-month longitudinal academic study of more than 640,600 free proxies from 11 providers, presented at NDSS MADWeb 2024, found that only 34.5% were ever active at all, identified 4,452 distinct vulnerabilities on proxy IPs — including 1,755 allowing remote code execution — and caught 16,923 proxies manipulating content in transit.

For anti-blocking specifically, free proxies are counterproductive: the IPs are shared by every other free-tier scraper hitting the same targets, so they arrive pre-burned on major sites' blocklists. You inherit other people's bad reputation on top of your own risk. The full argument is in our free versus paid proxies breakdown.

Monitor block signals continuously

Blocks rarely arrive all at once; they escalate. Instrument your scraper so you see the escalation early:

1. Status-code mix per domain. A rising share of 403, 429, or 503 responses is the first alarm. Alert on trends, not single failures.
2. CAPTCHA and challenge-page detection. Many sites return HTTP 200 with a challenge page instead of content. Fingerprint challenge pages (title patterns, known markup) and count them as failures, or your success metrics lie to you.
3. Response-size and content anomalies. A product page that suddenly returns 5 KB instead of 80 KB, or parses with zero extracted fields, is likely a block page or decoy data.
4. Latency shifts. Progressive throttling — responses slowing from 300 ms to 5 seconds — is a soft block signal that precedes hard bans.
5. Success rate per proxy subnet and per session. If failures cluster on specific IP ranges, retire those ranges; if they cluster on long-lived sessions, shorten your session lifespan.

When signals fire, the correct response is to slow down and diagnose — not to escalate aggression. I would treat a rising 429 share the way I treat a climbing error rate in a production system: stop pushing and find the cause. Halve your request rate, verify your headers still match a current browser, check whether the site changed its structure or its terms, and only then resume. Scrapers that respond to blocks by pushing harder convert temporary throttling into permanent bans.

What we deliberately will not cover

Some anti-blocking questions have answers we will not publish, because the technique only exists to defeat access controls:

• Bypassing paywalls or login walls. Content behind authentication is access-controlled by design. Automating around that control is unauthorized access, whatever the tooling.
• CAPTCHA solver farms. A CAPTCHA is a site saying no to automation. Industrial-scale solving exists to override that refusal.
• Scraping personal data. Names, emails, profiles, and contact details carry legal obligations (GDPR, CCPA) and real harm potential. Reputable providers block these use cases in their acceptable-use policies for good reason.
• Purchase automation. Sneaker bots, ticket scalping, and account farming are the use cases that got proxies a bad name and that provider KYC programs exist to filter out.

If a project needs any of these, it does not have a blocking problem; it has a permission problem, and no engineering fixes that.

Checklist: an ethical anti-blocking setup

• Read robots.txt and ToS; confirm the data is public and non-personal
• Prefer an official API or feed where one exists
• Rate limit per domain with jitter; obey Retry-After; back off exponentially
• Send a complete, internally consistent, current browser header set
• Persist cookies within sessions; pair cookie jars with IPs; expire sessions plausibly
• Use rotating residential IPs from a KYC-enforcing provider when volume genuinely requires it
• Match sticky-session windows to your longest stateful flow
• Monitor status codes, challenge pages, content anomalies, and latency per domain
• Respond to block signals by slowing down and diagnosing, never by escalating

Handled this way, anti-blocking is not an arms race — it is basic courtesy expressed in engineering. Sites block traffic that behaves badly; the durable fix is traffic that behaves well. For the broader context on tools, legality, and architecture, continue with our full web scraping guide.