Request-based scraping vs a headless browser
Request-based scraping and browser-based scraping collect the same data. They differ in what runs on your side and in what the anti-bot system gets to see.
- What runs: a request-based scraper is an HTTP client that sends the requests your code builds. A browser-based scraper drives a real browser through Playwright, Puppeteer or Selenium.
- Cost per page: an HTTP client does not render anything or execute the site's JavaScript, so it needs far less CPU and memory per page and can run many more sessions on the same machine. It also skips images, fonts and third-party scripts: in a residential proxy bandwidth benchmark by the Hyper Solutions team, a request-based scraper used about 22 times less proxy traffic per hotel search than a default headless browser.
- What the anti-bot system sees: a browser brings a genuine TLS handshake, genuine headers and a real JavaScript environment. An HTTP client has to reproduce the network-level part itself.
- Challenges: a browser runs the vendor's scripts on its own, but automation frameworks leave traces that bot protection looks for. An HTTP client cannot run those scripts, so the challenge answers (Akamai sensor data, DataDome slider solutions, Kasada's
/tlpayload, Incapsula'sreese84) have to be generated for it. - Maintenance: browser setups break on browser and framework updates. Request-based flows break when an anti-bot vendor changes its scripts or endpoints.
What anti-bot systems check in an HTTP client
Before any challenge comes into play, the request itself has to pass. The Hyper Solutions core requirements list five points that apply to every vendor:
- A browser-grade TLS client. Default libraries such as Python
requests,axiosor Go'snet/httphave a non-browser TLS fingerprint. - Exact browser header order, including the HTTP/2 pseudo-headers. DevTools lists headers in its own order, not the order they left the browser, so take the order from a proxy capture of the wire.
- Session consistency: one User-Agent, one TLS fingerprint, one IP address, one header order and matching client hints for the whole flow, backed by a working cookie jar.
- Sticky proxies. The IP address a challenge payload is generated for has to be the one the site sees. Rotating proxies break that.
- Matching Chrome versions across the User-Agent,
sec-ch-uaandsec-ch-ua-platform.
Then come the vendor checks: the _abck cookie and its sensor data on Akamai, the slider and interstitial on DataDome, the x-kpsdk headers on Kasada, and reese84, ___utmvc and the captcha block on Incapsula.
How a request-based flow is built
- Record the browser. Capture a real session through a recording proxy and save it as a HAR file.
- Find the dynamic values. Search the capture for every token, cookie and script path a request depends on, and find where each one first appears.
- Rebuild the requests with a browser-grade client, in the browser's order and with the browser's headers.
- Generate the challenge payloads for the anti-bot system on the site.
- Compare your script with the browser, request by request, until there is no difference left.
When a browser is the simpler choice
Request-based scraping pays off when volume, speed or infrastructure cost matter. A browser can be the better tool when:
- the flow depends on long-lived browser state: sessions that have to stay alive for a long time, logged-in accounts, or many steps that build up state;
- you only need a handful of pages, once, and the setup time outweighs the running cost;
- the data only exists after complex client-side interaction and there is no underlying API request to call;
- the site uses a protection you have no payload generator for;
- the flow changes so often that keeping a request-level copy up to date costs more than running a browser.
The four request-based guides show the complete flow per vendor: Akamai, DataDome, Kasada and Incapsula.