Web scraping is not illegal by itself: in the US and the EU, collecting publicly available, non-personal data is generally lawful. As I read the cases, risk enters when you touch personal data, copyrighted or database-protected content, login-gated pages, or enforceable terms of service. Legality turns on what you collect, how you access it, and where the data subjects are.
Key takeaways
- No US or EU statute prohibits web scraping as an activity; courts assess it under existing laws such as the CFAA, contract law, copyright, and data protection.
- The hiQ v. LinkedIn litigation established that scraping publicly available data likely does not violate the CFAA, yet hiQ still lost on breach of contract and settled.
- Under GDPR, publicly visible personal data remains personal data: you need a lawful basis, transparency, and minimization to scrape it.
- The EU DSM Directive (2019/790) created text-and-data-mining exceptions, but Article 4 lets rightsholders opt out in machine-readable form.
- Public, non-personal, non-gated data collected at reasonable request rates for legitimate purposes is the low-risk zone; logins, paywalls, and personal data are the tripwires.
Before anything else: this is not legal advice
This article is an educational overview; I write it as a working web engineer, not a lawyer, and it does not create any attorney-client relationship. Web scraping law is unsettled, jurisdiction-specific, and fact-specific. If your business depends on scraped data, retain counsel qualified in every jurisdiction where you operate and where your target sites and data subjects are located. What follows summarizes primary sources and major court decisions as of July 2026.
It is also worth stating my compliance position up front: ProxyFacts covers scraping for legitimate purposes only, such as price monitoring, SEO tracking, ad verification, market research, and AI training data collection. I do not publish guidance on bypassing logins or paywalls, scalping, account creation, or harvesting personal data.
The short answer: scraping is a method, not an offense
There is no statute in the United States, the European Union, or the United Kingdom that says "web scraping is illegal." Scraping is simply automated retrieval of web pages, the same HTTP requests a browser makes, issued by a program instead of a person. Search engines, price-comparison services, academic researchers, and archive projects all rely on it.
What the law regulates is not the method but the surrounding facts:
- Access — did you take data the site made public, or did you get past a technical or authentication barrier?
- Content — is the data factual and non-personal, or is it copyrighted work, a protected database, or personal data?
- Contract — did you affirmatively agree to terms that prohibit scraping, for example by creating an account?
- Conduct — did your crawler behave reasonably, or did it degrade the target service?
Every major scraping dispute of the last decade turns on one or more of these four questions.
Web scraping law in the United States
The CFAA: the "without authorization" question
The main federal statute invoked against scrapers is the Computer Fraud and Abuse Act of 1986, codified at 18 U.S.C. Section 1030. It penalizes accessing a computer "without authorization" or "exceeding authorized access." For years, plaintiffs argued that scraping a website against its wishes, for example after a cease-and-desist letter, was unauthorized access.
Two decisions narrowed that theory. In Van Buren v. United States (2021), the Supreme Court adopted what it called a "gates-up-or-down" reading: the CFAA targets those who bypass gates they are not entitled to pass, not those who misuse information they were allowed to reach. The case involved a police officer misusing database access, not scraping, but its logic reshaped scraping analysis.
The hiQ v. LinkedIn arc
The defining scraping case is hiQ Labs v. LinkedIn. The timeline matters because people often quote only one stage of it:
| Stage | What happened |
|---|---|
| 2017 | District court granted hiQ a preliminary injunction, ordering LinkedIn to stop blocking hiQ's scraping of public profiles |
| 2019 | Ninth Circuit affirmed: scraping publicly available data likely does not violate the CFAA |
| 2021 | Supreme Court vacated and remanded for reconsideration in light of Van Buren |
| 2022 | Ninth Circuit reaffirmed its holding on remand: the CFAA's "without authorization" concept does not apply to public websites that have no authentication gate |
| Late 2022 | On the merits, the district court found hiQ had breached LinkedIn's User Agreement; the parties settled with a consent judgment against hiQ |
The takeaway is double-edged. Scraping data that any browser can see without logging in is very likely outside the CFAA in the Ninth Circuit. But hiQ still lost the war: contract claims survived, and hiQ agreed to stop scraping LinkedIn. "Not a federal hacking crime" and "legally safe" are different things. Whenever hiQ is cited as blanket permission to scrape, the late-2022 contract judgment is the stage I point to first.
The same pattern continued in Meta Platforms v. Bright Data (No. 3:23-cv-00077, N.D. Cal.). In January 2024, the district court granted summary judgment for the proxy provider Bright Data on Meta's breach-of-contract claims, reasoning that scraping public Facebook and Instagram pages while logged out was not bound by terms that apply to logged-in account holders. Meta later dropped the case. Again the dividing line was authentication: logged-out access to public pages sits on the safer side of it.
Contract, copyright, and other US theories
Even where the CFAA does not apply, US scrapers face:
- Breach of contract. Browsewrap terms (a link in the footer) are weakly enforceable; clickwrap terms (you clicked "I agree", typically at account creation) are strongly enforceable. Scraping while logged in to an account you created is the highest-contract-risk configuration.
- Copyright. Facts and raw data are not copyrightable in the US under Feist Publications v. Rural Telephone (1991), but creative expression is: reviews, articles, photographs, and curated compilations. Copying prices is different from republishing product descriptions.
- Trespass to chattels and unjust enrichment. State-law claims occasionally succeed where scraping measurably burdens the target's servers, which is one more reason to rate-limit aggressively.
Web scraping law in the European Union
GDPR: personal data is the biggest tripwire
The General Data Protection Regulation (EU) 2016/679 applies whenever scraped data relates to an identifiable person: names, usernames, photos, employer, location, and in many cases combinations of otherwise innocuous fields. Three points routinely surprise newcomers:
- Public does not mean free to process. GDPR contains no exemption for publicly available data. Scraping a public profile is still "processing" that needs a lawful basis under Article 6, in practice usually a documented legitimate-interests assessment.
- Transparency obligations reach scrapers. Article 14 requires telling people you obtained their data indirectly, within a month, unless an exception applies. Poland's data-protection authority issued one of the earliest GDPR fines against a company that scraped public business-registry data and failed to notify the individuals in it. This is the trap I would flag first in any design review.
- Territorial scope is broad. Under Article 3, GDPR can apply to a US or Asian scraper with no EU office if it monitors or targets people in the EU. Fines can reach 20 million euros or 4 percent of global annual turnover under Article 83.
Data-protection authorities have also acted collectively: since 2023, joint statements from regulators including the UK ICO and Canada's OPC have warned that mass scraping of personal data from social platforms can constitute a reportable data breach for the platform and unlawful processing by the scraper. If your project touches personal data at all, treat GDPR compliance as a design requirement, not an afterthought.
The DSM Directive: Europe's text-and-data-mining exceptions
The Directive on Copyright in the Digital Single Market (EU) 2019/790 created two copyright exceptions directly relevant to scraping:
- Article 3 permits text and data mining (TDM) for scientific research by research organizations and cultural-heritage institutions, and rightsholders cannot contract out of it.
- Article 4 permits TDM by anyone, including commercial actors, unless the rightsholder has expressly reserved those rights "in an appropriate manner," which for online content means machine-readable form. Robots.txt directives and emerging opt-out standards are the common implementations.
Article 4 has become the legal backbone of commercial AI-training-data collection in Europe, and the opt-out mechanism is why respecting machine-readable signals is more than etiquette there: scraping opted-out copyrighted content for TDM loses the exception.
Database rights and terms of service in the EU
The EU also protects databases in two extra ways. The Database Directive 96/9/EC grants a sui generis right against extracting or re-utilizing a substantial part of a database that took substantial investment to build, with no US equivalent. And in Ryanair v. PR Aviation (C-30/14), the Court of Justice of the EU held that where a database is not protected by the directive, the site owner may restrict scraping contractually through its terms of use. In other words, in the EU either the database right or the contract can bite, so neither "it is just facts" nor "the database is unprotected" is a complete defense.
Universal principles that apply almost everywhere
Jurisdictions differ, but four questions predict scraping risk in nearly all of them.
| Factor | Lower risk | Higher risk |
|---|---|---|
| Access | Public pages, no login, no paywall | Behind authentication, paywalls, or technical blocks aimed at you specifically |
| Data type | Prices, availability, specs, rankings, aggregate stats | Personal data, full creative works, whole protected databases |
| Agreement | No account, browsewrap only, logged-out access | Clickwrap terms you accepted, scraping while logged in |
| Conduct | Rate-limited, cached, off-peak, identified crawler | High-volume requests that degrade the target service |
Engineer’s take (Hinata): I read this table as an architecture checklist, because three of the four factors are fixed before any code runs — which pages, which fields, which account state. Only conduct is decided at runtime, so it is the only row a rate limiter can rescue. If I were designing a scraping pipeline, I would make the low-risk column structurally unavoidable: no credential store anywhere in the system, a schema that simply has no fields for personal data, and throttling enforced in the queue rather than in each worker. Retrofitting that after a cease-and-desist costs far more than building it in from day one.
Three practical corollaries:
- Public vs. gated is the brightest line. Both the hiQ and Bright Data outcomes turned on it, and the CFAA analysis effectively starts there. Anything behind a login is a different legal universe; this site does not publish techniques for it.
- Personal data changes everything. The moment scraped records identify people, GDPR (and CCPA-style laws in the US) attach obligations regardless of how public the source was.
- Copyright follows expression, not effort. Extracting factual data points is generally safer than reproducing the text, images, or curated structure around them, and in the EU the database right adds a separate layer.
What compliant scraping looks like in practice
Teams that scrape at scale for legitimate purposes, such as price monitoring or SEO rank tracking, tend to converge on the same operating rules:
- Scrape only publicly accessible pages; never authenticate, and never bypass paywalls or technical access controls.
- Prefer official APIs or licensed data feeds where they exist and cover your need.
- Respect robots.txt and machine-readable TDM opt-outs, especially for EU-relevant content.
- Minimize: collect the fields you need, not entire pages, and exclude personal data unless you have completed a documented GDPR analysis.
- Rate-limit and schedule crawls so your traffic is negligible relative to the target's normal load.
- Keep records: what you collect, from where, under which legal analysis, and who signed off.
- Take cease-and-desist letters seriously; continuing after explicit revocation converts a gray area into a documented dispute.
Rule 6 is the one I would automate first: a collection log costs almost nothing to write, and it is the artifact that shows your purpose was legitimate when someone eventually asks.
For the technical side of running polite, reliable crawlers, see our guide on how to scrape without getting blocked; for the full picture of building a collection pipeline, start with the complete web scraping guide.
Why proxy providers now vet their customers
The infrastructure layer has internalized this legal landscape. The major residential-proxy providers no longer sell anonymously to anyone with a credit card, and their published compliance programs are a useful signal when choosing a provider:
- Bright Data limits residential-network access to verified companies that pass a human-reviewed KYC process, including corporate-email verification, a use-case description, and potentially an introduction call and government-issued ID, according to its published KYC policy.
- Oxylabs requires every customer to complete a KYC form at signup, applies risk-based escalation such as ID verification and compliance calls, and is a member of the Ethical Web Data Collection Initiative, per its KYC and safety policy.
- Decodo (formerly Smartproxy) runs automated fraud checks and KYC on every registration and actively blocks high-risk targets such as banking, government portals, and ticketing sites, according to its security and compliance documentation.
- IPRoyal runs KYC through the third-party verifier iDenfy; it is mandatory for static residential proxies and unverified accounts remain partially restricted, per its KYC policy.
The practical upshot: if your use case cannot survive a vendor's KYC questionnaire, that is a strong sign it would not survive a court's scrutiny either. Understanding what a residential proxy is and how these networks source their IPs is part of the same diligence.
Bottom line
Web scraping is legal by default and unlawful by circumstance. In the US, the hiQ v. LinkedIn arc and Van Buren pushed the CFAA away from public-data scraping, while contract and copyright claims remain fully alive. In the EU, GDPR governs any personal data you touch, and the DSM Directive's Article 4 TDM exception, with its machine-readable opt-out, frames commercial mining of copyrighted content. Wherever you operate, the low-risk formula is stable: public pages, no logins, no personal data, respect for opt-outs, gentle request rates, and documentation of why your purpose is legitimate. When in doubt, ask a lawyer before you ask a crawler.