Skip to main content

Request-based scraping

Concept/General

Also known as HTTP-based scraping, scraping without a headless browser

Request-based scraping fetches pages and APIs with plain HTTP requests instead of running a headless browser. Against bot protection it only works when every request looks like it came from a real browser and the anti-bot challenges are answered with generated payloads.

How to recognize it

How to recognize it
Client
An HTTP client such as tls-client or azuretls-client rather than Playwright, Puppeteer or Selenium
Symptom
403, 429 or challenge pages that Chrome on the same IP does not get

Request-based scraping vs a headless browser

Request-based scraping and browser-based scraping collect the same data. They differ in what runs on your side and in what the anti-bot system gets to see.

  • What runs: a request-based scraper is an HTTP client that sends the requests your code builds. A browser-based scraper drives a real browser through Playwright, Puppeteer or Selenium.
  • Cost per page: an HTTP client does not render anything or execute the site's JavaScript, so it needs far less CPU and memory per page and can run many more sessions on the same machine. It also skips images, fonts and third-party scripts: in a residential proxy bandwidth benchmark by the Hyper Solutions team, a request-based scraper used about 22 times less proxy traffic per hotel search than a default headless browser.
  • What the anti-bot system sees: a browser brings a genuine TLS handshake, genuine headers and a real JavaScript environment. An HTTP client has to reproduce the network-level part itself.
  • Challenges: a browser runs the vendor's scripts on its own, but automation frameworks leave traces that bot protection looks for. An HTTP client cannot run those scripts, so the challenge answers (Akamai sensor data, DataDome slider solutions, Kasada's /tl payload, Incapsula's reese84) have to be generated for it.
  • Maintenance: browser setups break on browser and framework updates. Request-based flows break when an anti-bot vendor changes its scripts or endpoints.

What anti-bot systems check in an HTTP client

Before any challenge comes into play, the request itself has to pass. The Hyper Solutions core requirements list five points that apply to every vendor:

  1. A browser-grade TLS client. Default libraries such as Python requests, axios or Go's net/http have a non-browser TLS fingerprint.
  2. Exact browser header order, including the HTTP/2 pseudo-headers. DevTools lists headers in its own order, not the order they left the browser, so take the order from a proxy capture of the wire.
  3. Session consistency: one User-Agent, one TLS fingerprint, one IP address, one header order and matching client hints for the whole flow, backed by a working cookie jar.
  4. Sticky proxies. The IP address a challenge payload is generated for has to be the one the site sees. Rotating proxies break that.
  5. Matching Chrome versions across the User-Agent, sec-ch-ua and sec-ch-ua-platform.

Then come the vendor checks: the _abck cookie and its sensor data on Akamai, the slider and interstitial on DataDome, the x-kpsdk headers on Kasada, and reese84, ___utmvc and the captcha block on Incapsula.

How a request-based flow is built

  1. Record the browser. Capture a real session through a recording proxy and save it as a HAR file.
  2. Find the dynamic values. Search the capture for every token, cookie and script path a request depends on, and find where each one first appears.
  3. Rebuild the requests with a browser-grade client, in the browser's order and with the browser's headers.
  4. Generate the challenge payloads for the anti-bot system on the site.
  5. Compare your script with the browser, request by request, until there is no difference left.

When a browser is the simpler choice

Request-based scraping pays off when volume, speed or infrastructure cost matter. A browser can be the better tool when:

  • the flow depends on long-lived browser state: sessions that have to stay alive for a long time, logged-in accounts, or many steps that build up state;
  • you only need a handful of pages, once, and the setup time outweighs the running cost;
  • the data only exists after complex client-side interaction and there is no underlying API request to call;
  • the site uses a protection you have no payload generator for;
  • the flow changes so often that keeping a request-level copy up to date costs more than running a browser.

The four request-based guides show the complete flow per vendor: Akamai, DataDome, Kasada and Incapsula.

Questions

What is request-based scraping?

Request-based scraping collects data by sending HTTP requests directly to a site's pages and APIs, without running a browser. On sites with bot protection the requests have to match a real browser's TLS handshake, headers and session, and the anti-bot challenges have to be answered with generated payloads.

Is request-based scraping faster than using a headless browser?

Usually, yes. An HTTP client does not render pages or run the site's JavaScript, so it needs far less CPU and memory per page. The trade-off is that the browser-level details and the challenge answers have to be reproduced in your code.

Why is my request blocked when the same URL works in Chrome?

Most often because the request differs from Chrome at the network level: a different TLS fingerprint, a header order no browser sends, client hints that do not match the User-Agent, or a missing cookie or token from an anti-bot challenge.

Do I need a headless browser to get past bot protection?

Not necessarily. If the client matches Chrome at the TLS and header level and the challenges are answered with generated payloads, plain HTTP requests work on websites protected by Akamai, DataDome, Kasada and Incapsula.

Further reading

Technical review: Hyper Solutions engineering, Nov 18, 2026