Web Scraping Without Getting Blocked: The Complete Guide

$ ls ./sections

Read the rules before you read the DOM
The signals that add up to “bot”
The fixes, in order of effort
When to stop hand-rolling and buy the bypass
Edge cases and troubleshooting

HTTP/2 403 Forbidden
server: cloudflare
cf-mitigated: challenge

<!doctype html><html><head><title>Just a moment...</title></head>
<body>Verifying you are human before you continue.</body>

Cloudflare returned that before your request ever reached the origin: a 403 wrapped around an interstitial that waits for a real browser to prove itself. Web scraping without getting blocked is mostly the craft of never summoning that page, which comes down to a stack of small signals you already control.

Read the rules before you read the DOM

Before the fingerprinting and the proxy budget, there is a file most scrapers never open. robots.txt sits at the root of nearly every domain, and it lists the paths the owner is asking automated clients to leave alone. Reading it programmatically is about thirty seconds of standard-library work.

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

# Ask before you fetch. If this returns False, you already have your answer.
allowed = rp.can_fetch("MyScraperBot/1.0", "https://example.com/catalog")

Honoring that file is more than manners. A Crawl-delay and a list of disallowed paths are the site telling you, in writing, what its infrastructure can absorb, and the same values feed straight into the delay logic you are about to build anyway. Ignore them and your traffic looks identical to the abuse the anti-bot vendor was paid to stop, which lands your ASN on a reputation list faster than any header trick can talk it back off.

Identify yourself while you are at it. A custom User-Agent carrying a project name and a contact URL costs nothing and gives an annoyed sysadmin a way to email you before they reach for a block rule. Cache the parsed robots.txt per host so you are not refetching it on every request, and re-read it now and then, because these files change.

The terms of service are the other half of the homework. Many sites allow scraping of public pages and forbid it behind a login, some forbid it entirely, and pulling personal data or copyrighted content carries legal exposure that no proxy rotates away. None of this is legal advice, and the specifics shift by jurisdiction and by what you do with what you collect. The pragmatic point survives on its own: modest request rates, off-peak timing, and respecting the signals a site sends are exactly the behaviors that also keep you under its blocking threshold.

Good citizenship is the cheapest camouflage there is.

The signals that add up to “bot”

One weak signal rarely gets you blocked. A modern anti-bot stack scores you across a dozen of them at once, and it is the accumulated total that finally trips the wall.

Start with the headers, since they are read first and cost nothing to fix. A fresh requests install still ships a User-Agent of python-requests/2.x, the most greppable string in the entire transaction. A real browser also sends an Accept, an Accept-Language, an Accept-Encoding, and a coherent block of Sec-Fetch-* headers, in a stable and predictable order. A request that omits half of them and scrambles the rest reads as scripted before the IP is even inspected.

Underneath the headers sits a layer Python’s standard client cannot disguise: the TLS and HTTP/2 fingerprint. How your client negotiates the handshake (its cipher list and extension order, hashed into a JA3 or JA4 value) and how it sequences HTTP/2 frames both differ between real Chrome and a requests session, and the serious stacks check that fingerprint against the browser your User-Agent claims to be. When the header says Chrome and the handshake says OpenSSL, that contradiction is the tell. Libraries like curl_cffi exist to mimic a browser’s TLS profile, and that is as far down that particular rabbit hole as this guide will drag you.

Timing gives you away next. People are erratic and slow; a naive script fires at a metronomic interval, often several requests a second, sometimes in perfectly parallel bursts no human hand could produce. That regularity is precisely the shape a rate-limiter is tuned to catch.

Then there is state. A browser that reaches a deep product URL has usually passed through the homepage, collected cookies, and carries a Referer; a scraper that hits the same URL cold, with an empty cookie jar and no referrer, advertises that it skipped the front door. Sessions are cheap to carry and awkward to fake after the fact.

Last is the address itself. An IP inside a known cloud ASN (AWS, OVH, Hetzner) opens the conversation already distrusted, because anti-bot vendors keep standing reputation scores on those networks. That reputation is the one signal your code cannot rewrite, which is where the free fixes run out and residential proxies start to matter.

The fixes, in order of effort

The cheapest fixes cost nothing but attention, so start there and reach for the wallet only when identity itself is the wall.

Your baseline is a requests.Session wearing believable headers, pacing itself, and obeying what the server tells it. A single Session reuses the connection and keeps a cookie jar across calls, which erases two of the signals from the previous section for free. Jittered delays keep the timing from looking mechanical. And when a response comes back 429, the correct move is to read its Retry-After header and wait exactly that long, instead of retrying into a throttle until it hardens into a permanent ban.

import time
import random
import requests

# One Session reuses the TCP connection and, crucially, keeps cookies.
# A fresh connection per request with an empty cookie jar is a tell by itself.
session = requests.Session()

# Look like a browser a human is actually driving. The default
# python-requests User-Agent is the first string a WAF greps for.
session.headers.update({
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/126.0.0.0 Safari/537.36"
    ),
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
})


def polite_get(url, max_retries=4):
    """Fetch one URL like a considerate client: reuse the session,
    pace the requests, and obey what the server tells you.
    Check robots.txt and the site's ToS before pointing this anywhere real."""
    for attempt in range(1, max_retries + 1):
        # Jittered pause BEFORE the call. A fixed sleep is a metronome;
        # real traffic is uneven, so randomize the gap every time.
        time.sleep(random.uniform(1.5, 4.0))

        resp = session.get(url, timeout=20)

        if resp.status_code == 200:
            return resp

        if resp.status_code in (429, 403):
            # Retry-After is seconds or an HTTP date; seconds is the usual 429.
            retry_after = resp.headers.get("Retry-After", "")
            try:
                wait = float(retry_after)
            except ValueError:
                wait = min(2 ** attempt, 30)   # missing/odd header: exp backoff
            wait += random.uniform(0, 1.5)      # jitter so retries aren't in lockstep
            print(f"[{resp.status_code}] backing off {wait:.1f}s (attempt {attempt})")
            time.sleep(wait)
            continue

        # Anything else (5xx, odd redirects): don't guess, surface it.
        resp.raise_for_status()

    raise RuntimeError(f"Gave up on {url} after {max_retries} attempts")


if __name__ == "__main__":
    r = polite_get("https://books.toscrape.com/")  # a sandbox built for this
    print(len(r.text), "bytes")

That client carries you through most sites that were only ever irritated by python-requests defaults. It lives wherever the rest of my scraping toolchain does, with credentials and URL lists pulled from the environment, never pasted into the file. If the content only appears after JavaScript runs, the next rung up is a real headless browser, Playwright or Puppeteer driving actual Chrome, which fixes the fingerprint problem at the cost of a great deal more RAM per worker.

What none of that fixes is a burned address. Once your datacenter IP trips a reputation threshold, every request behind it inherits the block, and no header or delay digs you back out; the only lever left is the address the target sees.

The proxy and API links below are affiliate links: sign up through one and the site earns a commission at no extra cost to you, and it changes none of the recommendations.

Building and maintaining your own rotation layer in 2026 is a poor use of engineering hours. Route your requests through a rotating residential gateway like Oxylabs’ pool and let their network eat the ASN bans, cycling the exit IP on every call so the reputation resets before it can pile up. A residential address reads as ordinary home broadband, which is trust a datacenter range cannot buy at any price. Which pool actually holds up under sustained load is a separate argument with real money riding on it, and the residential proxy shortlist is where I fight it out. Start on a provider’s trial gigabytes and throw your single worst target at it before you commit to a plan.

When to stop hand-rolling and buy the bypass

There is a point where the fingerprinting arms race stops being worth your salary. When the target sits behind a full managed anti-bot (Cloudflare Turnstile, DataDome, Akamai) and you have already sunk two afternoons into TLS profiles that shatter on the next Chromium release, the honest accounting says this is no longer a problem worth hand-solving.

Hand it to a managed scraping API like ScrapingBee or ZenRows, which fold the proxy rotation, the headless rendering, and the challenge-solving behind a single HTTP call: you POST a URL and get rendered HTML back. The landing pages sell that as magic; the invoice sells it as time, and time is the honest pitch, because paying per successful request costs less than an engineer’s week lost to a fingerprint you would have to re-win next quarter anyway. Which of the two wins on a given wall depends entirely on the wall, so the dedicated head-to-head is where I run them side by side and name the winner.

One caveat the API vendors stay quiet about: whatever you build has to run somewhere that stays awake. A scraper on your laptop dies the second the lid closes, so the job belongs on a box built to run bots around the clock, with a scheduler firing it and somewhere durable to keep its logs. Buying the bypass is triage, and a senior engineer triages without ego about it.

Edge cases and troubleshooting

The failures that cost the most time are the ones that do not look like failures.

The silent soft-block. A 200 OK is no proof of success. A tripped anti-bot frequently serves a page that renders cleanly and contains nothing you came for: an empty results grid, a truncated DOM, or in the meaner deployments, deliberately altered numbers meant to quietly poison your dataset. Assert on content rather than on the status line. Choose an element that must exist on a genuine page (a price node, a known selector, a minimum row count) and treat its absence as a block instead of a parse error.

Honeypot links. Some sites plant <a> tags no human can see (display:none, zero-size, or shoved far off-screen) purely to trap crawlers that follow every href. Follow one and you have identified yourself in a single request. Before you enqueue a URL, confirm the element is actually visible and honor rel="nofollow"; do not blindly walk every link in the DOM.

Session and cookie expiry. A sticky proxy session that outlives its TTL rotates the exit IP out from under a multi-step flow, and the target logs you out or flags the account mid-sequence. Match the session hold time to how long the flow genuinely takes, and re-authenticate the moment an IP-echo check shows the exit address has changed underneath you.

Geo blocks and geo-fenced content. A site that serves different HTML by country, or refuses everything outside its home market, will hand a US exit node a polite empty page while a local IP sees the full inventory. Set the country explicitly on the gateway, then confirm it with a fast IP-geolocation lookup before the first real request, never trusting whatever exit the pool defaulted to.

None of these throw an exception, which is exactly what makes them expensive. Wire the content assertion in first, log the status code and a hash of the response body on every fetch, and you will catch the soft-block the day it starts rather than the week your database quietly fills with nulls.

Read the rules before you read the DOM

The signals that add up to “bot”

The fixes, in order of effort

When to stop hand-rolling and buy the bypass

Edge cases and troubleshooting

Oxylabs

ScrapingBee

ZenRows

Best Residential Proxies for Web Scraping in 2026

Best Residential Proxy Reddit: What the Useful Threads Agree On

Bright Data vs Oxylabs: The Honest Comparison