Method
We probe status, not content. Nothing here reads a page body.
What one capture does
For each of 1000 domains we fetch robots.txt and check four agent affordances, then attempt one request as each of five callers. One append-only JSONL row per observation.
Rails
- No evasion. We do not solve CAPTCHAs, rotate identities to get around a block, or retry to defeat a rate limit. A block is the measurement.
- No impersonation. Vendor-token callers append a disclosure naming this project. They are labelled simulations everywhere they appear, and that is a stated limitation of the instrument, not a footnote.
- We obey robots. Where it denies a caller we do not send that caller's request, and we report that separately from a refusal.
- No bodies, no cookies, no header maps. Rows keep policy and response metadata only.
Denominators
The headline reachability rate is over requests sent. Dividing by all attempts would fold our own compliance into the web's hostility and make the web look more hostile than it is.
Comparability
This capture declares vantage class `unrecorded`, robots-unavailable policy `unrecorded`, robots.txt redirect policy `unrecorded` and observation slot `unrecorded`. Captures differing on any of those were taken by different instruments, and the repository withholds a cross-capture delta until every comparability dimension matches. Where a dimension reads `unrecorded` the manifest predates it, and `unrecorded` never counts as agreement with another `unrecorded`.
The repository refuses to emit a cross-capture delta unless vantage class, observation slot, input list, row schema, robots-unavailable policy, probe-target redirect policy, robots.txt redirect policy, denial gate and caller set all match. An unrecorded dimension never counts as agreement with another unrecorded dimension.
The probe follows up to five consecutive redirects for robots.txt, as RFC 9309 requires, and applies the policy it finds in the context of the domain we asked about. It follows none for the page it is measuring: that destination has not been checked against robots itself. Those two policies are recorded separately, so relaxing one cannot hide behind the other.