Skip to main content
Education/General

Why Rotating Proxies and User Agents Can Get Your Scraper Blocked Faster

Rotating the IP and User-Agent on every request is standard advice for scraping without getting blocked. On modern anti-bots it often backfires, because they judge a session, and a session that changes identity on every request contradicts itself. Rotate between sessions, not inside them.

10 min read
On this page
General

You bought residential proxies, built a list of User-Agents and rotate both on every request, because that's what most advice on web scraping without getting blocked says to do. You're still blocked, and maybe sooner than before. That isn't bad luck. Modern anti-bot systems judge a session, not a single request, and a session whose IP, User-Agent and fingerprint change from one request to the next contradicts itself. Rotation helps between sessions. Inside a session, consistency is what passes.

Proxies and User-Agents are only part of how you bypass bot detection. This post explains why per-request rotation backfires, which contradictions it creates, and how to rotate in a way that works with the anti-bot instead of against it.

Key Takeaways

  • Anti-bots tie trust to a session: cookies and tokens are issued to the IP, User-Agent and fingerprint that earned them. Change one mid-session and the session no longer matches its own cookies.
  • Random User-Agent rotation creates contradictions a server can see: a Firefox User-Agent on a Chrome TLS handshake, Chrome client hints next to a Safari User-Agent, a sec-ch-ua that no real Chrome ever sent.
  • Rotate between sessions: a new session gets a new sticky IP, a fresh cookie jar and one internally consistent identity, and keeps them until it ends.
  • Good residential proxies still matter. When they get blocked anyway, the problem is usually the session riding on them, not the IPs.

Why per-request rotation backfires

Most anti-bot flows follow the same pattern. Your first requests look like a new visitor. The protection runs its checks, and when they pass, it hands that visitor something: an Akamai _abck cookie that becomes valid, a datadome cookie, an Incapsula reese84 token. From then on, it expects that visitor to keep looking like the visitor it trusted.

Per-request rotation breaks that in two ways.

The cookie no longer fits the traffic. A token earned on IP A with Chrome on Windows comes back on the next request from IP B with a different User-Agent. Several of these systems generate their payloads for one specific IP and User-Agent. On Akamai, a payload generated for one IP stops matching the moment the IP changes, and the session built on it is no longer valid.

Every request starts from zero. If each request arrives as a new visitor, each one meets the anti-bot as an unknown client and has to pass its checks from scratch. You pay for those checks again and again, in time and in solves, and never get the benefit of a session that has already earned trust.

Rotating proxies make the first problem worse in a way that is easy to miss. A rotating proxy hands out a new IP per connection, not per session you define. The IP you read at the start of a flow is not necessarily the IP the site sees three requests later. Our proxies and IP docs walk through how that breaks a payload: you fetch your IP, generate a payload for it, the proxy rotates when it opens the next connection, and the site sees a different address than the one the payload was built for.

The contradictions random User-Agent rotation creates

A User-Agent string is one claim among many. A real browser's claims agree with each other, and a randomly picked string from a list usually disagrees with something:

  • User-Agent versus TLS fingerprint. A Firefox or Safari User-Agent on a connection whose TLS handshake is Chrome's (or Python's) says two different things before the first header is read.
  • User-Agent versus client hints. Chrome on desktop and Android sends sec-ch-ua, sec-ch-ua-mobile and sec-ch-ua-platform on every HTTPS request. Firefox and Safari don't send these hints at all, and neither does Chrome on iOS, which runs on WebKit. A Safari User-Agent next to sec-ch-ua, or a desktop Chrome User-Agent without it, describes no real browser.
  • User-Agent versus platform. A Windows User-Agent with sec-ch-ua-platform: "macOS" is a contradiction in two headers of the same request.
  • User-Agent versus header order. Browsers send HTTP/2 pseudo-headers in different, stable orders: Chrome sends :method :authority :scheme :path, Firefox and Safari use other orders. A Firefox User-Agent with Chrome's order disagrees with itself.
  • User-Agent versus the calendar. Since Chrome 153 in September 2026, Chrome ships a new stable version every two weeks. A User-Agent list collected a year ago is mostly versions nobody runs anymore, and a version that hasn't been released yet is just as obvious.

The client hint is the one people get wrong most often, because it looks like a version number you can template. It isn't a template, though it is deterministic: Chromium derives the placeholder brand, its version and the brand order from the major version. Here is what Chrome 156 and Chrome 157 actually send:

Chrome 156: "Not:A-Brand";v="8", "Chromium";v="156", "Google Chrome";v="156"
Chrome 157: "Not-A.Brand";v="99", "Google Chrome";v="157", "Chromium";v="157"

The placeholder brand, its version and the order of the brands all change between versions. Take the 156 value and bump the numbers to 157, and you get a sec-ch-ua that no Chrome has ever sent. A User-Agent rotator that only swaps version numbers produces exactly that on every request. Derive the value with Chromium's algorithm for that major version, or capture it from a real browser.

What has to stay the same inside a session

Think of a session as one visitor. From its first request to its last, keep:

  • One IP, from a sticky proxy, and the same IP in any payload request you make for it.
  • One User-Agent, on a current Chrome version, with sec-ch-ua, sec-ch-ua-mobile and sec-ch-ua-platform that match it, and one platform: don't switch between Windows and macOS mid-session.
  • One HTTP client and TLS profile, so every request has the same handshake (TLS fingerprinting).
  • One header order per request type, captured from a real browser (header order).
  • One cookie jar, updated only by the server, without duplicates.
  • The anti-bot's tokens, carried forward as the site issued them.
  • The request sequence a browser would make: the page before its API calls, the challenge before the protected request.

None of these is exotic. They are the core requirements we list for every integration, and they all describe the same thing: one browser, behaving like one browser, for as long as the session lasts.

Per-request rotation versus per-session rotation: every request a different visitor holding someone else's cookies, versus one consistent identity per session.
Rotation inside a session turns every request into a different visitor holding someone else's cookies. Rotation between sessions gives each visitor one consistent identity.

Rotate between sessions, not inside them

Rotation is still useful. It spreads load across IPs and lets you walk away from an identity that has gone bad. The unit you rotate is the whole session.

Start a new session when:

  • The IP is hard-blocked. On DataDome, a block page whose dd object has t set to bv means the IP itself is banned; solving won't help, so move the session to a new IP.
  • The session can't be recovered. The anti-bot keeps rejecting it after you have handled its challenges correctly.
  • The session has done its job, or your proxy's sticky session is about to expire.

A new session means a new sticky IP, a fresh cookie jar and a new HTTP client, with one identity chosen up front. You can vary that identity between sessions, as long as each one is internally consistent. We deliberately don't give a number of requests per session: it depends on the site and the vendor, and our powhttp guide shows how to measure it on your own target by replaying a request until a session stops being trusted.

Many residential proxy providers let you pin a sticky session with an ID in the proxy username, and a new ID gets a new IP. In a hotel-search scraper we built, each session put its own session ID into the Massive proxy username with a 30-minute session TTL, longer than any session ran. The exact syntax differs per provider.

In code, the point is to create the identity once and let everything in the session read from it:

import uuid
from dataclasses import dataclass, field

import tls_client

# Sticky-session syntax differs per proxy provider; this template is a placeholder.
PROXY_TEMPLATE = "http://user-session-{sid}:pass@proxy.example:8000"

CHROME_157_WINDOWS = {
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
    "sec_ch_ua": '"Not-A.Brand";v="99", "Google Chrome";v="157", "Chromium";v="157"',
    "platform": '"Windows"',
}


@dataclass
class ScrapeSession:
    """One visitor: one sticky IP, one identity, one client and cookie jar."""

    identity: dict = field(default_factory=lambda: dict(CHROME_157_WINDOWS))
    sid: str = field(default_factory=lambda: uuid.uuid4().hex[:12])

    def __post_init__(self):
        self.proxy = PROXY_TEMPLATE.format(sid=self.sid)
        self.client = tls_client.Session(
            client_identifier="chrome_152",  # newest profile; Chrome's TLS is unchanged from 152 to 157
            random_tls_extension_order=True,
            disable_http3=True,
        )
        self.client.proxies = {"http": self.proxy, "https": self.proxy}

    def base_headers(self):
        return {
            "sec-ch-ua": self.identity["sec_ch_ua"],
            "sec-ch-ua-mobile": "?0",
            "sec-ch-ua-platform": self.identity["platform"],
            "user-agent": self.identity["user_agent"],
        }


session = ScrapeSession()
# ... run the whole flow with session.client and session.base_headers() ...

# Hard block (for example DataDome t='bv') or the session has done its job:
# throw the whole session away and start a new one. Never swap only the IP or UA.
session = ScrapeSession()

The identity, the proxy session and the client are created together and thrown away together. Nothing in the flow can change one of them on its own.

Residential proxies still blocked? Check the session first

Good residential proxies matter. For some sites, datacenter and known proxy ranges are a detection signal on their own, and no client fix makes a flagged address clean. But a residential IP can't fix a session that contradicts itself, and that is the more common reason they "stop working".

To tell the two apart, put the exact proxy your scraper uses into a real browser and load the same page:

  • The browser is blocked too: it's the IP. Start a new session on a different one.
  • The browser gets through and your scraper doesn't, on the same IP: it's your session. Look for rotation inside the session, a mismatched User-Agent and client hints, the wrong TLS profile or a broken cookie jar.

And check the proxy setup itself: sticky mode, a session lifetime longer than your flow, a location that makes sense for the site, and the same exit IP in your payload requests as in your requests to the site. You can read your exit IP from https://ip.hypersolutions.co/ip, called through the same proxy.

Common advice, and what to do instead

  • "Rotate the IP on every request." Use one sticky IP per session and rotate when you start a new session.
  • "Rotate User-Agents from a big list." Use one current Chrome User-Agent per session, with matching client hints, platform and TLS profile.
  • "Clear cookies so you look new." Keep one clean jar per session; when you want to look new, start a new session.
  • "Residential proxies fix bot detection." They fix IP reputation. The session has to be consistent on top of them.

When the session is consistent and the site still challenges it, the remaining step is the anti-bot's own check, and that one is vendor-specific. Our request-based guides cover it for Akamai, DataDome, Incapsula and Kasada.

FAQ

Should I rotate User-Agents when scraping?

Not per request. Pick one current, real Chrome User-Agent per session, with client hints, platform and TLS profile that match it, and keep it for the whole session. Vary the identity between sessions if you want, as long as each one is consistent.

Why do residential proxies still get blocked?

Usually because of what rides on them: a session whose IP, User-Agent, client hints or TLS fingerprint change or disagree, or a cookie jar that carries tokens from another identity. Test the same proxy in a real browser. If the browser passes, the proxy is fine and the problem is your session.

Sticky or rotating proxies for scraping?

For anti-bot protected sites, sticky. The IP has to stay the same for the whole session so the anti-bot's cookies and payloads keep matching your traffic. Rotate by starting a new session on a new sticky IP.

How often should I rotate proxies?

There is no universal number. Rotate when a session is hard-blocked, can't recover, has finished its work or reaches the end of its sticky lifetime. How many requests a session survives depends on the site and the vendor; measure it on your own target instead of copying a number.

Can better proxies and User-Agents bypass bot detection?

Not on their own. Proxies and User-Agents are two of the signals. Anti-bots also read the TLS handshake, header order, client hints, cookies and the order of your requests, and many run a challenge on top that only a browser, or a payload generated for one, can answer.

Next steps

Try it yourself

Solve Akamai, DataDome, Incapsula and Kasada in one request

Skip the browser. Generate the sensor data, tokens and cookies you just read about over plain HTTP, with a free one-week trial.