Web Scraping Guide 2026: The Complete Business Handbook

Web scraping is the automated extraction of publicly available data from websites. Businesses use it for price monitoring, SEO tracking, ad verification, market research, and AI training data. Done lawfully, it combines public-data-only collection, compliant infrastructure such as residential proxies or scraping APIs, and respect for target sites. In this guide I walk through the whole stack, end to end.

What web scraping is (and is not)

Web scraping means using software to request web pages the way a browser does, then parsing the returned HTML, JSON, or rendered DOM into structured data: prices, product attributes, search rankings, ad placements, job listings, review text. The output typically lands in a database or data warehouse where analysts and models consume it.

It helps to separate three terms that get blurred:

1. Crawling — discovering pages by following links, the way search engines index the web. Crawling answers "what pages exist?"
2. Scraping — extracting specific fields from known pages. Scraping answers "what does this page say right now?"
3. Data APIs — official, sanctioned endpoints a site offers. Always prefer an official API when one exists and covers your need; scraping is the fallback for the majority of the web that offers no API.

What web scraping is not, at least on this site: it is not a tool for buying up sneakers or concert tickets, farming accounts, bypassing paywalls or logins, or harvesting personal data. ProxyFacts covers infrastructure for lawful, public-data collection only, and every provider we track now actively polices these abuse categories — more on that in the compliance section below.

How a scraping pipeline works

A production scraping pipeline has five stages, and understanding them explains most of the buying decisions later in this guide:

1. URL sourcing — sitemaps, category pages, or search results define what to fetch.
2. Fetching — an HTTP client or headless browser requests each page. This is where proxies, retries, and anti-bot handling live, and it is where nearly all the cost and failure risk concentrates.
3. Parsing — selectors or LLM-based extractors turn raw HTML into structured records.
4. Validation — schema checks, deduplication, and freshness rules catch silent breakage when a site changes its layout.
5. Storage and delivery — the clean dataset flows to a warehouse, dashboard, or training corpus.

Everything a proxy provider or scraping API sells you targets stage two. The rest stays your responsibility no matter what you buy. If I had to name the stage teams underinvest in, it is validation: fetch failures page you loudly, but a silent layout change just fills your warehouse with garbage.

Legitimate business use cases in 2026

Demand for scraped data keeps growing. Proxyway's independent Proxy Market Research 2026 (data collected March–April 2026) reports Bright Data at roughly $300M annualized revenue growing 50% year over year, driven largely by AI demand. That growth rests on a handful of well-established, lawful use cases:

• Price and product monitoring. Retailers and brands track competitor prices, stock levels, and assortment across marketplaces, often hourly. Accuracy depends on seeing the same localized prices a real shopper sees, which is why this workload leans on residential IPs — see our dedicated guide to proxies for price monitoring.
• SEO and SERP tracking. Agencies and in-house teams monitor keyword rankings, featured snippets, and local pack results across countries and cities. Search engines aggressively rate-limit automated queries, making this one of the most proxy-dependent workloads — covered in depth in proxies for SEO monitoring.
• Ad verification. Advertisers verify that campaigns actually display where and how they paid for, and detect ad fraud, by viewing pages as real users in specific locations. Our guide to ad verification proxies explains the geo-targeting requirements.
• AI training data. Collecting public web text, images, and metadata for model training and retrieval corpora became the industry's biggest growth driver. The licensing and consent questions are evolving fast — we track them in proxies for AI training data.
• Market and academic research. Job-posting analytics, review mining, real-estate trends, and news aggregation all run on scraped public data.

If your use case is not on a list like this — if it involves personal data, credentialed access, scarce-inventory purchasing, or circumventing a paywall — the reputable end of the proxy industry will refuse the business, and so does our coverage.

Is web scraping legal? The short version

The honest answer: scraping publicly available, non-personal data is generally lawful in the US, EU, and UK, but "generally" is doing real work in that sentence. Legality turns on three questions:

1. What are you collecting? Public facts (prices, rankings, availability) sit on the safest ground. Copyrighted creative works and personal data carry substantially more risk. GDPR and CCPA apply to personal data whether or not it is publicly visible.
2. How are you collecting it? Scraping behind a login typically means you accepted terms of service, which changes the contract analysis entirely. Anonymous access to public pages is the pattern that has fared best in litigation.
3. Where do you and the target operate? Jurisdiction matters — outcomes under the US Computer Fraud and Abuse Act, EU database rights, and national implementations of GDPR differ.

Court decisions over the past decade have repeatedly distinguished public-data scraping from unauthorized access, but the landscape keeps moving, and terms-of-service, copyright, and privacy claims remain live risks even where anti-hacking statutes do not apply. We maintain a full, regularly updated analysis in is web scraping legal? — read it before starting any new collection project. I am an engineer, not a lawyer, so treat nothing on this site as legal advice.

Beyond the strictly legal question sits an ethical baseline that also happens to reduce blocking and business risk: identify a legitimate purpose, take only what you need, keep request rates low enough that the target never notices you operationally, and honor robots.txt and explicit opt-outs wherever practicable.

The proxy layer: why you need it and what to use

A scraper that sends thousands of requests from one IP address looks nothing like organic traffic, and websites respond with rate limits, CAPTCHAs, and blocks. Proxies solve this by routing requests through intermediate IP addresses, distributing your traffic so each individual IP behaves plausibly.

Residential vs. datacenter proxies

The two workhorse proxy types differ in where their IPs come from, and that difference drives everything else:

A residential proxy routes your request through an IP address assigned by a consumer ISP to a real household device — the full mechanics, including how providers source those IPs ethically through consented SDK partnerships, are explained in what is a residential proxy?. Because protected sites treat consumer IPs far more leniently, residential pools command premium prices: the four major providers we track claim pools of 64M+ (IPRoyal), 115M+ (Decodo), 175M+ (Oxylabs), and 400M+ monthly IPs (Bright Data) — all vendor claims as displayed on their product pages in July 2026, not independently verified counts.

Datacenter proxies remain the right default for targets that do not fight automation: they are faster and dramatically cheaper. Most mature scraping operations run a tiered setup — datacenter first, residential as the fallback for blocked or geo-sensitive requests. Our full breakdown of when each type wins is in residential vs. datacenter proxies.

Two smaller categories round out the market: ISP (static residential) proxies, which are datacenter-hosted IPs registered under consumer ISPs — useful when you need a stable residential-looking IP for days at a time — and mobile proxies, which route through cellular carrier IPs and are the most trusted and most expensive tier.

Why free proxies are not an option

Every few months someone asks why they should pay when free proxy lists exist. The evidence against free proxies is unusually strong. A 30-month academic study presented at NDSS MADWeb 2024, "Free Proxies Unmasked", analyzed more than 640,600 free proxies from 11 providers and found that only 34.5% were ever active; the researchers identified 4,452 distinct vulnerabilities on proxy IPs — including 1,755 permitting remote code execution — and caught 16,923 proxies manipulating content in transit. Independent industry outlet Proxyway reaches the same conclusion: free proxies routinely intercept traffic, inject content, and disappear without notice.

For business data collection, a free proxy is not a cost saving; it is an unaudited man-in-the-middle handling your traffic. The full risk analysis, including what "free" actually costs in failed requests, lives in free vs. paid proxies.

Build vs. buy: raw proxies or a scraping API?

Once you accept that you need paid infrastructure, the next fork is architectural: buy proxy access and build the fetching layer yourself, or buy a scraping API that handles fetching end to end.

The two models

Raw proxy access gives you an endpoint and credentials; your code handles sessions, retries, header and TLS fingerprinting, JavaScript rendering, and CAPTCHA encounters. You pay per GB (residential) or per IP (datacenter), whether requests succeed or fail. Costs are lowest per request at scale, and you keep full control.

Scraping APIs (also sold as web unlockers or scraper APIs) accept a URL and return the page content or parsed data. The provider handles proxy selection, browser rendering, fingerprinting, and unblocking internally, and — critically — you typically pay only for successful results. You trade per-request cost and some control for reliability and near-zero maintenance.

What the pricing actually looks like

Published entry pricing for the scraping-API model, from each provider's own pricing pages as of July 17, 2026:

• Oxylabs Web Scraper API: from $0.25 per 1K successful results; the free trial includes up to 2K results with no credit card, per Oxylabs' product page.
• Decodo Web Scraping API: a permanent free plan with 2,000 requests per month, then a $19/mo Core plan at $0.50 per 1K requests (standard proxies); premium-proxy and JS-rendering requests cost $1.50 per 1K, prices plus VAT, per Decodo's pricing.
• Bright Data Web Unlocker: a free tier of 5K requests per month, pay-as-you-go at $1.50 per 1K requests, and a $499/mo Scale plan including 383K requests, per Bright Data's pricing page.
• IPRoyal Web Scraping API: a newer, early-access product from $1.00 per 1K requests including rendering, rotation, and CAPTCHA solving, per IPRoyal's product page.

How to decide

Choose raw proxies plus in-house tooling when:

• You have engineers who can own the fetching layer as a product.
• Your targets are mostly low-protection sites where success rates stay high without unlocking tech.
• Volume is large enough that per-GB pricing beats per-request pricing decisively.

Choose a scraping API when:

• Targets deploy serious anti-bot systems and you would otherwise burn engineering weeks on fingerprinting.
• Paying only for successful results matters more than the lowest theoretical unit cost.
• The team consuming the data is analysts or data scientists, not infrastructure engineers.

The decision framework, with worked cost scenarios, is in scraping API vs. proxy, and our comparison of the leading API products is in best scraping APIs. In practice the answer is often "both": route hard targets through an API and everything else through cheaper raw proxies. If I were standing this up from zero, I would start every new target on the API and only migrate it to raw proxies once its success rate had stayed boringly stable for a few weeks.

Staying unblocked — ethically

Anti-bot systems do not primarily exist to stop lawful data collection; they exist to stop fraud, credential stuffing, and infrastructure abuse. An ethical scraper's goal is to look like what it is — a polite, low-impact automated visitor — not to defeat security controls protecting accounts or payments. Within that frame, the techniques that keep success rates high are well documented:

1. Throttle aggressively. Concurrency limits and randomized delays per domain are the single biggest factor. If your scraper measurably loads a target's servers, you are over the line operationally regardless of legality.
2. Rotate IPs sensibly. Rotating proxies assign a fresh IP per request; sticky sessions hold one IP for minutes or hours when a multi-step flow needs continuity. All four major providers support both — Oxylabs and Decodo document sticky sessions up to 24 hours and multi-day windows respectively, and IPRoyal advertises sticky intervals up to 7 days, per their July 2026 product pages.
3. Send coherent fingerprints. Headers, TLS signatures, and browser behavior must match. A Chrome user-agent with a Python TLS fingerprint is an instant flag.
4. Render only when required. Headless browsers consume far more compute than plain HTTP requests. Many sites serve the data you need in embedded JSON without rendering.
5. Respect the site's signals. Honor robots.txt where practicable, back off on 429 responses, cache aggressively, and scrape during the target's off-peak hours.
6. Monitor success rates per target. Silent degradation — blocks that return plausible-looking but wrong content — is the failure mode that poisons datasets.

What we will not document: techniques for evading security on login-protected systems, defeating purchase limits, or masking identity for fraud. The line is simple — access control on public pages is an arms race the industry openly participates in; access control on accounts and payments is a security boundary. The full playbook, with implementation details, is in how to scrape without getting blocked.

Choosing your infrastructure provider

The proxy market consolidated around a handful of serious providers, and in 2026 they differentiate less on pool size claims and more on pricing structure, tooling, and compliance posture. Here is how the four providers we track compare on published facts, all taken from their own pricing and policy pages on July 17, 2026:

Engineer’s take (Hinata): The row I would read first is KYC approach, not price — a provider that cannot onboard you makes every other cell irrelevant, and Bright Data's companies-only gate will exclude many solo builders outright. At small volume the per-GB spread also matters less than it looks: below roughly 10 GB a month, the gap between $4 and $7.35 is lunch money next to one engineer-day lost to debugging blocks. What no pricing table shows is that metered billing punishes mistakes — a misconfigured retry loop burns paid gigabytes on failed fetches, so I would set a per-target spend alert before the first production run. And if your workload is genuinely irregular, IPRoyal's non-expiring traffic changes the shape of the cost, not just the rate.

A few notes the table cannot carry:

• Oxylabs pairs its residential product (sticky sessions up to 24 hours, HTTP3 and SOCKS5 support, and vendor-claimed 0.41 s average response time and 99.95% success rate) with the market's lowest published scraping-API entry price. Details in our Oxylabs review and pricing at oxylabs.io.
• Bright Data claims the largest pool and runs the strictest gate: residential network access requires a human-reviewed KYC limited to verified companies, including corporate email verification and potentially an intro call and government ID, per its KYC FAQ. Full analysis in our Bright Data review, and the head-to-head with Oxylabs in Bright Data vs. Oxylabs.
• Decodo is Smartproxy's new identity — the rebrand went official on April 22, 2025, with accounts, pricing, and endpoints carried over unchanged, per Decodo's own announcement. It remains the value pick with small-plan granularity (3 GB at $11.25/mo, +VAT) and a published ethical sourcing policy. See our Decodo review.
• IPRoyal differentiates on never-expiring pay-as-you-go traffic — useful for irregular workloads where a monthly subscription wastes gigabytes — but has no standard free trial and a narrow refund policy, per its official FAQ. See our IPRoyal review.

The selection framework

Match the provider to the workload, in this order:

1. Compliance fit first. Can you pass the provider's KYC, and does your use case survive their acceptable-use policy? If a provider would reject your use case, that is information about the use case.
2. Targeting requirements. City or ZIP-level geo targeting is table stakes for price monitoring and ad verification; ASN targeting matters for network research.
3. Session model. Rotating for stateless collection, sticky for multi-page flows. Check the maximum sticky window against your longest flow.
4. Pricing shape. Steady high volume favors committed tiers ($2.00–$2.50/GB at the top plans); spiky or experimental workloads favor pay-as-you-go or IPRoyal's non-expiring traffic. Our proxy pricing comparison tracks the current per-GB math across all four.
5. Trial before commitment. Decodo's 3-day trial plus 14-day money-back window is the lowest-risk on-ramp; Oxylabs and Bright Data offer trials with more process; IPRoyal effectively asks for a $7.35 test purchase.

The complete checklist — including SLA questions, support benchmarks, and red flags — is in how to choose a proxy provider, and our ranked recommendations for the residential category specifically are in best residential proxies.

The 2026 market: what changed

Three shifts define the proxy market this year, per Proxyway's Proxy Market Research 2026, which benchmarked 13 providers with data collected in March–April 2026:

1. Prices stabilized. After residential rates fell by as much as 75% between 2023 and 2025, the decline flattened. Oxylabs and IPRoyal retired long-running discount codes and instead revised permanent plans down roughly 25% — cleaner list pricing, less coupon theater.
2. Pools kept growing. The median advertised residential pool size reached 54M IPs across benchmarked providers — with the usual caveat that these are marketing claims, not audited counts.
3. AI demand reshaped the customer base. Bright Data's cited ~$300M annualized revenue, growing 50% year over year on AI demand, illustrates where the growth is: training-data and retrieval pipelines now sit alongside classic price-intelligence workloads.

One more trend worth flagging: Proxyway also documents grey-market entry pricing under $0.50/GB. Treat those offers the way you treat free proxies — sourcing you cannot verify is risk you cannot price. The compliant end of the market moved in the opposite direction, with universal KYC (Oxylabs and Decodo screen every customer at signup; Bright Data restricts residential access to verified companies) and active blocking of high-risk targets like banking and ticketing, per each provider's published policies.

How ProxyFacts evaluates this market

A disclosure and a methodology note. ProxyFacts is monetized through affiliate partnerships, always disclosed where they appear. This guide contains no affiliate placements. We have not yet completed our own performance benchmarks; until we have, every number on this site comes from provider documentation, published pricing pages, or independent research, each attributed with its source and retrieval date, and vendor performance claims (response times, success rates, pool sizes) are labeled as claims. When our own test data exists, it will replace vendor claims and be clearly marked as first-party measurement.

Our compliance stance is equally simple: we cover infrastructure for lawful collection of public, non-personal data. We do not publish guidance for sneaker or ticket bots, account creation at scale, paywall or login circumvention, or personal-data harvesting — and we favor providers whose policies enforce the same lines.

Where to go next

This guide is the hub; the spokes go deeper on every decision it introduced:

• Understand the legal ground rules: is web scraping legal?
• Learn the infrastructure: what is a residential proxy? and residential vs. datacenter proxies
• Make the build-vs-buy call: scraping API vs. proxy, then best scraping APIs
• Keep success rates high: how to scrape without getting blocked and free vs. paid proxies
• Pick and price a provider: how to choose a proxy provider, proxy pricing comparison, and best residential proxies
• Apply it to your use case: price monitoring, SEO monitoring, ad verification, and AI training data

Start with the legality guide if you are scoping a new project, or the provider-selection guide if your pipeline already exists and the fetching layer is the bottleneck.