feat: store browser / HTTP client fingerprint in Session#3646
Merged
Conversation
…able Previously, calling retire() bumped errorScore to maxErrorScore but a subsequent markGood() (e.g. the automatic markGood after a successful requestHandler that explicitly retired the session) could decrement the score back below the threshold, making the session usable again. Track retirement in a dedicated _retired flag checked by isUsable() so retire() is a true terminal state.
Replace the global EVENT_SESSION_RETIRED listener and the per-controller browserSessionIds map with a check at the per-request cleanup hook: if the session ended the request unusable, retire the browser controller. The previous mechanism tore down browsers eagerly mid-flight; the new one lets the in-flight request finish on the doomed browser and retires it once the request is done. Same outcome, no global event subscription needed.
SessionPool no longer extends EventEmitter and no longer fires a sessionRetired event. The Session->SessionPool back-reference, the sessionPool constructor option on Session, and the EVENT_SESSION_RETIRED constant are gone with it. The only consumer of that event was the browser crawler, which now retires browsers via the per-request context pipeline cleanup. Custom createSessionFunction implementations that manually constructed Session instances should drop the sessionPool argument.
Define a small, dependency-free fingerprint contract on the Session interface so both HTTP clients and browser crawlers can read the same shape. The rich browser fingerprint payload lives in an opaque `browserFingerprint` slot to keep `@crawlee/types` free of any dependency on `fingerprint-generator`.
Expose a `fingerprint` field on `Session` (constructor option, getter, setter, and persisted in `getState()`) so HTTP clients and browser crawlers can read and write the session's browser/HTTP client fingerprint. Hydration via `SessionPool._maybeLoadSessionPool` rounds back through the same option, so persisted fingerprints survive restarts.
When a session is attached to the launch context, the prelaunch fingerprint hook now reads from and writes to `session.fingerprint` as the source of truth, falling back to the existing LRU only when the session has no fingerprint yet. Generated payloads are written back to `session.fingerprint.browserFingerprint` together with derived `userAgent`/`headers`/`browser`/`platform`/`device`/`locales` fields, so HTTP backends and persisted sessions can consume the same shape.
`BaseHttpClient.sendRequest` now resolves `session.fingerprint` next to `proxyUrl`/`cookieJar` and forwards it to `fetch` via the `CustomFetchOptions` interface, so concrete clients can apply the session-declared fingerprint on a best-effort basis.
`ImpitHttpClient.fetch` now consumes `session.fingerprint` when present: `fingerprint.browser` is mapped to impit's `Browser` enum (chrome and firefox supported, others ignored), and `fingerprint.headers` plus `fingerprint.userAgent` are merged under the actual request headers (request wins) so impit's browser impersonation defaults get overridden to match the session.
Drop the dedicated `fingerprint` field on `CustomFetchOptions` in favor of forwarding the whole `session` so concrete clients can read any session-declared state (fingerprint, future fields) without us growing the interface each time. `proxyUrl` and `cookieJar` stay as the *effective* values after `SendRequestOptions` override resolution — both are now documented in JSDoc as overrides to be preferred over reading `session.proxyInfo`/`session.cookieJar` directly. `ImpitHttpClient` now reads `options.session?.fingerprint` instead of `options.fingerprint`.
Drop `userAgent`, `headers`, `httpVersion`, and `locales` from `SessionFingerprint`. Header-level details already travel with the `Request` itself and can be overridden through the regular crawler options; HTTP-client backends that impersonate a browser derive them from `browser` anyway, so duplicating them on the session created two sources of truth. The interface now carries only request-shape hints (`browser`, `platform`, `device`) plus the opaque `browserFingerprint` slot. `ImpitHttpClient` no longer merges fingerprint headers into the request; it only maps `fingerprint.browser` to impit's `Browser` option. `browser-pool`'s `deriveSessionFingerprint` writes back only the narrow fields.
The opaque slot on `SessionFingerprint` is renamed from `browserFingerprint` to `details` to reflect that it can carry any consumer-specific payload corresponding to the lean hints — not just a browser fingerprint. `@crawlee/browser-pool` continues to stash a `BrowserFingerprintWithHeaders` there, but the slot no longer presumes that contents.
Revert the `session` field on `CustomFetchOptions` in favor of passing `fingerprint?: SessionFingerprint` directly. The JSDoc now frames it as *suggestions* about what browser-like setup the request should impersonate (browser/platform/device plus opaque `details`); concrete clients apply what they can and ignore the rest on a best-effort basis. `ImpitHttpClient` reads `options.fingerprint` again.
… version Bump impit from ^0.9.0 to ^0.14.0. The plain `chrome` / `firefox` aliases that impit accepts as browser identifiers fall back to the oldest available impersonation profile (chrome100, firefox128), which is itself a fingerprinting signal. Pick one of the concrete versioned profiles at random instead, and cache the choice per session fingerprint (via a `WeakMap` keyed on the fingerprint object) so a given session keeps impersonating the same browser version across its requests rather than rerolling every call. Adds `impit: true` to `pnpm-workspace.yaml`'s `allowBuilds` map for pnpm 11 (the package has a native postinstall).
`SessionPool._invokeCreateSessionFunction` now seeds a default
{browser, platform, device} fingerprint when none is supplied. The
`platform` matches the host OS (windows/macos/linux from
`process.platform`, mirroring `browser-pool`'s
`getGeneratorDefaultOptions`); `browser` and `device` are picked
uniformly at random from the realistic profiles for that platform
(e.g. android only with chrome/firefox mobile, no edge or safari on
linux). Pool-wide and per-call options still override the default, and
fingerprints rehydrated from KVS state arrive through `perCallOptions`
so they take precedence over the seed.
`SessionFingerprint` is now strictly read-only intent: `browser`, `platform`, `device`. The opaque `details` slot is gone — consumers keep their own derived state instead of stashing it on the session. `browser-pool`'s prelaunch hook stops writing back to `session.fingerprint` and no longer reads `details`. The existing session-keyed LRU is now the canonical cache for the rich `BrowserFingerprintWithHeaders`; on a miss, the generator is fed defaults overlaid with the session's hints (so a session with `browser: 'firefox'` always gets a firefox fingerprint, regardless of pool-wide defaults). Trade-off: a session hydrated from KVS gets a fresh rich payload in the same browser/platform/device family — exact replay of the original userAgent string does not survive a restart. That keeps the session as a pure intent declaration and removes a question of who owns the cache.
janbuchar
approved these changes
Jun 3, 2026
| } | ||
|
|
||
| /** | ||
| * Build a {@apilink SessionFingerprint} whose `platform` matches the host OS |
Contributor
There was a problem hiding this comment.
Is there a good reason to try to match the host OS? Is it not possible to impersonate a completely different one well enough?
Member
Author
There was a problem hiding this comment.
This is legacy Crawlee v3 behaviour
crawlee/packages/browser-pool/src/fingerprinting/utils.ts
Lines 42 to 55 in ab34886
With e.g. impit, you can emulate any OS well enough; it's not the case with browsers (e.g., system fonts leak the OS, and there isn't much you can do about it).
Also, since the interface is now explicitly best-effort, I wouldn't see this as much of an issue, really ⬇️
Contributor
There was a problem hiding this comment.
This is legacy Crawlee v3 behaviour
…
the interface is now explicitly best-effort, I wouldn't see this as much of an issue, really
These two points actually make me think that the default fingerprint should just not mention the OS at all and let the "consumer" decide.
Alternatively, we could rotate the OS so that there can be more variety in the generated fingerprints. And the consumer is still free to ignore the hint if they don't know how to impersonate a particular OS.
Member
Author
There was a problem hiding this comment.
Fair, you're right that the OS-stickyness is ratherBrowserPool's problem. I dropped the host-OS check here, let's see if this breaks anything
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
fingerprintfield toSessionso repeated requests with the same session can stay consistent on user-agent, headers, and TLS profile across HTTP clients and browser crawlers.Closes #3628.