Skip to main content

Python requests returns 403 Forbidden: you call requests.get(url) and get <Response [403]>, while the same URL opens fine in your browser, and maybe curl even returns the page. A 403 is a signal, not a diagnosis. Before you change headers, rotate proxies or rewrite code, read what the server sent back: the status, headers, cookies and body usually tell you whether the site refused you, your IP is flagged, your HTTP client gave itself away, or an anti-bot such as Akamai, DataDome, Incapsula or Kasada wants a challenge solved.

This post shows how to read that response, what each answer means for a Python scraper, and where to go next.

Key Takeaways

  • Read the 403 first. The response tells you which layer refused you, and each layer has a different fix.
  • Anti-bot vendors leave observable signals: a DataDome dd object, Akamai _abck cookies, Incapsula reese84 or incap_ses_* cookies, Kasada x-kpsdk-* headers. Use them for triage, not as proof.
  • requests cannot look like Chrome. It speaks HTTP/1.1, has a Python TLS fingerprint and puts its own default headers first. A Chrome User-Agent fixes none of that.
  • "curl works" means the difference is in your client, not your IP. It doesn't mean curl looks like a browser.
  • Fix in order: the IP, then the client, then the challenge. A payload cannot rescue a request that does not look like a browser.

Read the 403 before you change anything

A 403 Forbidden in web scraping comes from one of four places:

  • The application. You are not logged in, the API wants a key, or the content isn't available in your region. The body is the site's own error page or a JSON error. That is not a bot block, and no header will change it.
  • The IP. The address you are coming from is flagged. A real browser on that IP is blocked too.
  • Your HTTP client. The request doesn't look like a browser: wrong TLS handshake, wrong HTTP version, wrong headers.
  • The anti-bot challenge. The request looks fine, but the session hasn't passed the vendor's checks yet.

So print everything the server sent back before you touch your code:

import requests

resp = requests.get(url, timeout=30)
print(resp.status_code, resp.reason)
for name, value in resp.headers.items():
    print(f"{name}: {value}")
print(resp.text[:1000])

Which anti-bot is it?

Match what you see against the signals below. They are the same ones our own tooling uses to tell the four vendors apart. Treat them as triage: they show which protection is on the site, and some sites run more than one, so a vendor's cookies being present doesn't prove that vendor sent this particular 403.

  • DataDome. A 403 block page with a dd object that loads a script from ct.captcha-delivery.com (i.js for the interstitial, c.js for the slider), a datadome cookie, x-datadome or x-dd-* headers. On a JSON endpoint the challenge can arrive inline as a body like {"url": "https://geo.captcha-delivery.com/..."}. If the dd object has t set to bv, the IP itself is hard-blocked and solving won't help. Identified DataDome? Continue with the DataDome guide.
  • Akamai Bot Manager. Cookies like _abck and bm_sz, plus bm_so or sbsd_o on sites that use SBSD. The status code says less here: a session without valid sensor data can load the page normally and only fail on the login, search or add-to-cart call. A 429 whose body is only {"t":"..."} is an SBSD challenge, and a 428 is SEC-CPT. A different Akamai 403 is the edge deny: Server: AkamaiGHost and an "Access Denied" page saying you don't have permission to access the URL on this server, with a Reference #18.xxxxxxxx id. That is a WAF or edge rule, not the _abck or SBSD flow, and there is no challenge on the page to solve. Fix the IP and the client first. Identified Akamai? Continue with the Akamai guide.
  • Incapsula / Imperva. A reese84 cookie, incap_ses_* and visid_incap_* cookies, an x-iinfo header, a script under /_Incapsula_Resource, or a "Pardon Our Interruption" page. The classic Incapsula block page says "Request unsuccessful. Incapsula incident ID" and usually loads an iframe from /_Incapsula_Resource?.... Identified Incapsula? Continue with the Incapsula guide.
  • Kasada. x-kpsdk-* headers and a reference to an ips.js script. Kasada usually answers with a 429, not a 403. Identified Kasada? Continue with the Kasada guide.

Cloudflare is not one of the four, but you will meet it often, and its headers are easy to misread. Every response from a site behind Cloudflare carries server: cloudflare and a cf-ray header, blocked or not, so those two only tell you the site uses Cloudflare. The rest of the response tells you which case you are in:

  • A challenge. A cf-mitigated: challenge header and a "Just a moment..." interstitial. Cloudflare wants its JavaScript challenge passed before it serves the page, and a plain HTTP client can't run it. The client fixes below are a prerequisite, not a solution.
  • A hard block. The "Attention Required! | Cloudflare" page with "Sorry, you have been blocked", often with an error code such as 1020. A firewall rule on the site matched your request. There is nothing to solve: change what the rule matched, which is usually the IP, the country or the client.
  • Neither. A 403 with server: cloudflare and cf-ray but no challenge or block page most likely came from the site itself. Read the body as you would without Cloudflare in front.

Not every block is a 403. If you only check for 403, you will misread the 429 and 428 responses above.

A small helper that collects the signals for you:

def antibot_signals(resp, session_cookies=None):
    """Triage helper: which anti-bot signals does this response carry?

    Returns {vendor: [evidence, ...]}. An empty dict means no known signal.
    It shows what is present, not what caused the block.
    """
    body = resp.text[:50_000].lower()
    headers = {name.lower() for name in resp.headers}
    server = resp.headers.get("server", "").lower()
    cookies = {c.name.lower() for c in resp.cookies}
    if session_cookies is not None:
        cookies |= {c.name.lower() for c in session_cookies}

    found = {}

    def hit(vendor, evidence):
        found.setdefault(vendor, []).append(evidence)

    if "captcha-delivery.com" in body:
        hit("DataDome", "captcha-delivery.com in body")
    if "datadome" in cookies:
        hit("DataDome", "datadome cookie")
    if any(h.startswith(("x-datadome", "x-dd-")) for h in headers):
        hit("DataDome", "x-datadome / x-dd-* header")

    for name in ("_abck", "bm_sz", "bm_so", "sbsd_o"):
        if name in cookies:
            hit("Akamai Bot Manager", f"{name} cookie")

    if "akamaighost" in server:
        hit("Akamai edge / WAF", "Server: AkamaiGHost")
    if "access denied" in body and "reference #" in body:
        hit("Akamai edge / WAF", "Access Denied page with Reference #")

    if "reese84" in cookies:
        hit("Incapsula / Imperva", "reese84 cookie")
    if any(c.startswith(("incap_ses_", "visid_incap_")) for c in cookies):
        hit("Incapsula / Imperva", "incap_ses_* / visid_incap_* cookie")
    if "x-iinfo" in headers:
        hit("Incapsula / Imperva", "x-iinfo header")
    if "_incapsula_resource" in body:
        hit("Incapsula / Imperva", "_Incapsula_Resource script or iframe")
    if "incapsula incident id" in body:
        hit("Incapsula / Imperva", "Incapsula incident ID block page")

    if any(h.startswith("x-kpsdk-") for h in headers):
        hit("Kasada", "x-kpsdk-* header")
    if "ips.js" in body:
        hit("Kasada", "ips.js reference")

    # server and cf-ray are on every Cloudflare response: they only mean
    # the site is behind Cloudflare, not that Cloudflare blocked you.
    if server == "cloudflare" or "cf-ray" in headers:
        hit("Cloudflare", "server: cloudflare / cf-ray (site is behind Cloudflare)")
    if resp.headers.get("cf-mitigated", "").lower() == "challenge":
        hit("Cloudflare", "cf-mitigated: challenge header")
    if "just a moment..." in body:
        hit("Cloudflare", '"Just a moment..." challenge page')
    if "attention required! | cloudflare" in body or "sorry, you have been blocked" in body:
        hit("Cloudflare", '"Sorry, you have been blocked" block page')

    return found


if resp.status_code in (403, 428, 429):
    print(resp.status_code, antibot_signals(resp) or "no known anti-bot signal")

resp.cookies only holds the cookies that one response set, so pass session.cookies on later requests in a session. If the result is empty, or only says the site is behind Cloudflare, read the body yourself. It is either the application, or a protection the helper doesn't know: the client fixes below still apply, and if you want to know whether we cover it, ask us in Discord before you build anything around it.

Rule out the IP

The fastest test we know: put the exact proxy your scraper uses into a browser (a per-browser proxy extension is enough) and load the same URL.

  • If the browser is blocked too, it's the IP. Change the proxy. For some sites, datacenter and known proxy ranges are a detection signal on their own, and no code change fixes a flagged address.
  • If the browser gets through and your script doesn't, on the same IP, the problem is in your request. The rest of this post is about that case.

Find the layer first. Each one has a different fix, so work top to bottom:

You seeLayerWhat you do
The site's own error: a login prompt, JSON error or region messageApplicationNot a bot block. Fix auth, permissions or region
A real browser is blocked too, on the same proxy IP and URLIPChange the proxy or IP. No code change fixes a flagged IP
Only your script is blocked: python-requests UA, HTTP/1.1, Python TLSClientLook like Chrome on the wire: TLS profile, header order, one session
A browser-grade request is still blocked: a dd object, an _abck or reese84 cookie, x-kpsdk-* headersChallengeSend the payload a browser would. Continue with that vendor's guide
No known signal at allOtherRead the body yourself, and check support before building around it

What Python requests actually sends

This is the request requests.get(url) puts on the wire, captured on requests 2.34 with a raw socket server (host shortened to example.com):

GET / HTTP/1.1
Host: example.com
User-Agent: python-requests/2.34.2
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive

The User-Agent gives it away, so the standard advice is to set a Chrome one, usually together with a few headers copied from DevTools:

requests.get(url, headers={
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
    "sec-ch-ua": '"Not-A.Brand";v="99", "Google Chrome";v="157", "Chromium";v="157"',
    "accept-language": "en-US,en;q=0.9",
})

And this is what that sends:

GET / HTTP/1.1
Host: example.com
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
sec-ch-ua: "Not-A.Brand";v="99", "Google Chrome";v="157", "Chromium";v="157"
accept-language: en-US,en;q=0.9

The User-Agent now says Chrome. Everything around it still says Python:

  • HTTP/1.1. Chrome uses HTTP/2 wherever the server supports it. requests sends HTTP/1.1, and urllib3's HTTP/2 support is still experimental and opt-in.
  • requests' defaults stay first. Your headers are merged into the session defaults, so Accept-Encoding, Accept and Connection keep their slots and your headers are appended after them. Chrome's order looks nothing like this, and Chrome's accept-encoding is gzip, deflate, br, zstd. requests sends gzip, deflate, and urllib3 adds br or zstd only when the brotli or zstandard package is installed.
  • Mixed case. Title-Case defaults next to lowercase headers copied from DevTools. On HTTP/1.1 Chrome's casing is fixed per header: User-Agent and Accept are Title-Case, the sec-ch-ua client hints are lowercase. On HTTP/2 every header is lowercase. A mix copied from DevTools matches neither.
  • Missing headers. No sec-ch-ua-mobile, sec-ch-ua-platform, sec-fetch-* or upgrade-insecure-requests. A real Chrome navigation sends all of them, plus priority on HTTP/2 and HTTP/3.
  • Python's TLS handshake. Before any header is read, the TLS ClientHello has already identified the client. Python's ssl module offers different cipher suites and extensions than Chrome, and Chrome has shuffled its extension order on every connection since Chrome 110, so a fixed order stands out too. A custom SSLContext on an HTTPAdapter can change the cipher suites and curves, and with them the JA3/JA4, but ssl can't reproduce Chrome's extensions, GREASE values or shuffled order. You can make the handshake different; you can't make it Chrome.

That last point is why no amount of header work makes requests pass a serious anti-bot. The header says Chrome; the connection says Python.

Why curl works but Python requests doesn't

The answer is almost never that curl looks like a browser. It doesn't. Three differences explain most cases:

  • A different TLS fingerprint. curl builds its handshake with its own TLS library and settings; requests gets its handshake from Python's ssl module. A rule aimed at the fingerprint of Python HTTP clients can let curl through, and the reverse happens too. Neither looks like Chrome.
  • HTTP/2. The curl command-line tool has negotiated HTTP/2 by default for HTTPS since version 7.47.0, when it is built with HTTP/2 support. requests sends HTTP/1.1 by default. If the site's rules weigh the protocol, that alone can split the results.
  • The copied command carries more than you port. "Copy as cURL" in DevTools includes every header the browser sent, cookies included. Translated to requests, the defaults get mixed in and the order changes. And the cookies you paste were issued to another session: the browser's IP, User-Agent and fingerprint. Replaying _abck, datadome or reese84 from a different client and IP is a session change the anti-bot can see.

So "curl works" tells you something useful: the endpoint isn't blocking every non-browser client, and since curl and your script share the same IP, the difference is in your client. It does not mean shelling out to curl, or switching to pycurl, is a fix. On an endpoint the site protects properly, curl is blocked as well.

Fix the client: Chrome's TLS fingerprint and header order

To get past fingerprinting you need an HTTP client that reproduces Chrome's TLS handshake and lets you control header order, including the HTTP/2 pseudo-headers. In Python, these are the clients our own examples use:

  • tls-client, a Python wrapper around the Go library bogdanfinn/tls-client. Use the maintained Nintendocustom fork. Its API is close to requests, so it is the smallest change from code you already have.
  • rnet, an async client with browser emulation profiles (Emulation.Chrome*), if your scraper is built on asyncio.

Configure either one the same way: the latest Chrome profile your version of the library ships, random TLS extension order enabled, HTTP/3 disabled (most proxies don't support it yet), and one cookie jar in one session object for the whole flow.

Then send Chrome's headers in Chrome's order. This is a top-level navigation, the request Chrome makes when you type a URL into the address bar:

import tls_client

PROXY = "http://user:pass@proxy.example:8000"  # a sticky session, not a rotating one

session = tls_client.Session(
    client_identifier="chrome_152",  # the newest chrome_* profile your version ships
    random_tls_extension_order=True,
    disable_http3=True,
)
session.proxies = {"http": PROXY, "https": PROXY}

# HTTP/2 pseudo-headers, in Chrome's order
session.pseudo_header_order = [":method", ":authority", ":scheme", ":path"]

# Regular headers, in the order Chrome sends them on a navigation.
# "cookie" sits before "priority" even when there are no cookies yet:
# otherwise the client appends cookies at the end and priority is no longer last.
session.header_order = [
    "sec-ch-ua",
    "sec-ch-ua-mobile",
    "sec-ch-ua-platform",
    "upgrade-insecure-requests",
    "user-agent",
    "accept",
    "sec-fetch-site",
    "sec-fetch-mode",
    "sec-fetch-user",
    "sec-fetch-dest",
    "accept-encoding",
    "accept-language",
    "cookie",
    "priority",
]

headers = {
    "sec-ch-ua": '"Not-A.Brand";v="99", "Google Chrome";v="157", "Chromium";v="157"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": '"Windows"',
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/157.0.0.0 Safari/537.36",
    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "sec-fetch-site": "none",
    "sec-fetch-mode": "navigate",
    "sec-fetch-user": "?1",
    "sec-fetch-dest": "document",
    "accept-encoding": "gzip, deflate, br, zstd",
    "accept-language": "en-US,en;q=0.9",
    "priority": "u=0, i",
}

resp = session.get("https://www.example.com/", headers=headers)
print(resp.status_code)

Chrome 157 was the current stable version when we wrote this. Send whatever is current when you read it, and keep the User-Agent, sec-ch-ua and sec-ch-ua-platform on the same version and platform. The TLS profile can lag a few versions behind when your library hasn't shipped the newest one yet, because Chrome's ClientHello doesn't change with every release. Treat that as a temporary fallback.

Three rules catch most people right after they switch clients:

  • Header order is per request type. A navigation, a fetch() call to a JSON API, a script load and the anti-bot's own endpoint each have their own headers and order. A fetch() call has no upgrade-insecure-requests or sec-fetch-user, for example. Capture the exact request you are replaying.
  • Never copy header order from DevTools. It re-sorts headers for display. Capture the order from the wire instead.
  • Never set Content-Length yourself. The client computes it, and setting it manually can send it twice.

Keep the session consistent

Anti-bots judge the session, not a single request. From the first request to the last, keep these the same:

  • The IP, from a sticky proxy. Rotating proxies hand out a new IP per connection, and an anti-bot cookie issued to one IP no longer fits traffic from another. Rotate between sessions, not inside one.
  • The cookie jar, updated only by the server. Never send the same cookie name twice with different values; a broken jar is a common cause of 403s that come and go.
  • The User-Agent and its client hints. The Chrome version in the User-Agent, sec-ch-ua and sec-ch-ua-platform must agree. sec-ch-ua is more than a number: its brand list and order change between versions, so bumping only the numbers produces a value no real Chrome sent.
  • A current Chrome version. Chrome has shipped a new stable version every two weeks since Chrome 153 in September 2026, so a hardcoded version goes stale fast. Wait a few days after a release, then update the User-Agent, sec-ch-ua and sec-ch-ua-platform together.
  • Client hints only when asked. sec-ch-ua, sec-ch-ua-mobile and sec-ch-ua-platform go on every request. Higher-entropy hints such as sec-ch-ua-full-version-list only go out after the site asked for them in an Accept-CH response header.

And don't retry a 403 in a tight loop. The same flawed request doesn't look more like a browser the tenth time, and it can get a clean IP flagged.

Check what actually went out on the wire

Your code is not proof of what your client sent. Route the script through a debugging proxy that records the real wire (header order, case, HTTP/2 pseudo-headers and the TLS handshake), record the same request from a clean Chrome profile, and diff the two until the diff is empty. We use powhttp because it barely changes the TLS fingerprint it intercepts; our powhttp guide walks through the whole loop, including the browser-versus-script test over the same proxy.

If you'd rather not compare header lists by eye, export the capture as a HAR and run Check my HAR in our Discord, or let the Claude Code plugin read it through powhttp's MCP server.

When a browser-grade request still gets a 403

Once your TLS fingerprint, headers, IP and cookies match a real Chrome, the 403s that remain are different in kind. The protection isn't objecting to how the request looks anymore. It is waiting for proof that only a browser running its JavaScript can produce:

  • Akamai wants sensor data posted until _abck is valid, and on some sites an SBSD payload or a SEC-CPT proof-of-work as well.
  • DataDome wants its interstitial or slider solved, and expects the browser's telemetry (tags) along the way.
  • Incapsula / Imperva wants a reese84 token, a ___utmvc cookie, or both.
  • Kasada wants a payload posted to its /tl endpoint, and a fresh proof-of-work header (x-kpsdk-cd) on every protected request.

No header produces these. A headless browser runs the vendor's scripts and produces them as a side effect, at the cost of a browser per session plus the bandwidth and infrastructure that come with it. Or you keep the request-based setup and generate the payloads with an API.

That second option is what Hyper Solutions does. Your code keeps owning every request to the target site; our API never talks to it. You send the API the inputs a challenge needs (the vendor script, the relevant cookies, your IP and your User-Agent), it returns the payload a real browser would have produced, and you post that payload to the site yourself, from the same session. The Python SDK installs with pip install hyper-sdk.

A valid payload sent from a session that fails the earlier sections is still blocked: the payload says Chrome, and the connection has to agree. Each vendor guide above walks through its flow with Python code, and the Akamai, DataDome, Incapsula and Kasada product pages cover what each API solves.

Checklist for a Python requests 403

  • Printed the full response and identified the layer: application, IP, client, challenge or unknown
  • Checked 428 and 429 responses as well as 403
  • Tested the proxy IP in a real browser on the same URL
  • Replaced requests with a Chrome-profile TLS client: latest profile, random extension order, HTTP/3 off
  • Header order and pseudo-header order captured from the wire for each request type, with cookie before priority
  • User-Agent, sec-ch-ua and sec-ch-ua-platform on the same, current Chrome version
  • One sticky IP, one cookie jar and one User-Agent per session
  • Script and browser diffed until the diff is empty
  • The vendor's challenge handled with its own flow

FAQ

Why does Python requests return 403 when the page works in my browser?

Because the server can tell the two apart before it reads your User-Agent. Your browser sends a Chrome TLS handshake over HTTP/2, with a fixed set of headers in a fixed order, and it runs the site's anti-bot JavaScript. requests sends a Python TLS handshake over HTTP/1.1 with its own default headers first, and runs no JavaScript at all. On a protected site, that difference is enough for a 403.

How do I know which anti-bot is blocking my requests?

Read the response. DataDome serves a block page with a dd object and scripts from captcha-delivery.com; Akamai sets _abck and bm_sz cookies and uses 429 and 428 for SBSD and SEC-CPT, and its edge deny is an AkamaiGHost "Access Denied" page with a Reference #; Incapsula sets reese84 or incap_ses_* cookies, and its block page shows an "Incapsula incident ID"; Kasada sends x-kpsdk-* headers and usually answers 429. Cloudflare marks a challenge with cf-mitigated: challenge and a hard block with its "Sorry, you have been blocked" page; server: cloudflare and cf-ray alone only mean the site is behind Cloudflare. Those signals tell you which protection is on the site, which is the starting point, not proof of what caused this particular block.

Why does curl work but Python requests gets a 403?

curl and requests have different TLS fingerprints, and the curl tool negotiates HTTP/2 by default while requests uses HTTP/1.1, so a rule can block one and let the other through. That doesn't mean curl looks like a browser: on a properly protected endpoint, both are blocked. Treat "curl works" as a sign that the problem is your client rather than your IP.

Does setting a User-Agent fix a 403 in Python requests?

Only against the simplest filters, the ones that look for python-requests in the User-Agent. Anti-bots also check the TLS handshake, the HTTP version and the header order, and a Chrome User-Agent on a Python connection is a contradiction they can see.

Is every 403 Forbidden a bot block?

No. Missing authentication, a missing API key, account permissions and regional restrictions all return 403 as well, usually with the site's own error page or a JSON error. Bot blocks carry the vendor's signals listed above.

Can I fix a 403 with the requests library alone?

Sometimes, if the site only filters on obvious headers. If it runs a real anti-bot, no: requests can't make its TLS fingerprint match Chrome's, sends HTTP/1.1 by default, and can't produce the challenge payloads these systems require.

Next steps

Try it yourself

Solve Akamai, DataDome, Incapsula and Kasada in one request

Skip the browser. Generate the sensor data, tokens and cookies you just read about over plain HTTP, with a free one-week trial.