About the Aiprobase research crawler

User-agent
Mozilla/5.0 (compatible; Aiprobase-Research/1.0; +https://www.aiprobase.com/research/crawler)

We operate a small research crawler that studies how organisations publish machine-readable information about themselves — structured data, agent-facing files, and entity links. We publish what we find, together with the full methodology.

This page exists so that anyone who sees Aiprobase-Research in their server logs can find out exactly who we are, what we collect, and how to stop us.

On this page
  1. What it does
  2. What it collects
  3. What we do with it
  4. How to stop it
  5. How to verify it is really us
  6. Contact

What it does

For each domain in a study sample, our crawler initiates at most 14 requests — or at most 16 if your homepage redirects to another host of your domain, such as www, not counting redirects on robots.txt (see below).

We request them in a fixed order:

  1. /robots.txt on your domain
  2. your homepage — we handle its redirects ourselves, see below
  3. if the homepage redirects to another host of your domain (for example from example.com to www.example.com): the /robots.txt of that host, then the homepage there. From this point on, everything is requested from that host, and its robots.txt applies. "Your domain" means the same registrable domain, such as example.co.uk; we change host at most once
  4. seven small service files: /llms.txt, /llms-full.txt, /agents.md, /.well-known/ucp, /api/ucp/mcp, /mcp, /api/mcp
  5. an "about" page — /about, then /about-us if the first does not exist — and a "contact" page — /contact, then /contact-us
  6. at most one browser render of the homepage — see below

We read the robots.txt of every host before requesting anything else from it. If robots.txt redirects, we follow up to 5 redirects within your domain, as RFC 9309 asks; these steps are counted separately from the limits above. If the chain is longer or leads to a different domain, we treat robots.txt as unavailable and do not crawl your site. Your homepage may take at most 3 redirects in total; each counts towards the limits above, and the last attempts at the about and contact pages are dropped to make room. A longer chain and we treat the homepage as unreachable. We do not follow a redirect to a different domain. The service files are small text files that describe your site to machines; we request them on the host where your homepage actually lives, so that a file published only on www is not missed.

We do not crawl beyond these. Product pages, service pages and anything reached through your sitemap are out of scope.

It sends at most one request every two seconds per domain, and visits each domain once per pass. A baseline study is a single pass. Studies run roughly twice a year.

If your homepage is refused with 401, 403 or 429, we record the refusal and stop: nothing else is requested from your domain. If another page is refused, we do not retry it or switch to a different client to get around it.

Rendering. If the static fetch of your homepage returns no structured description of your organisation, we load that one page again in a real browser, in case the markup is added by scripts. Where the first fetch failed on a network error, the browser load is the only one. The browser loads the final homepage address found above, sends the same user-agent and does not download images, fonts, media files or stylesheets. If the page tries to navigate to a different domain, the browser stops.

One thing worth stating plainly: when a page is rendered, its own scripts make their own requests to your server — XHR and fetch calls written by you, not by us. We do not block those, because markup injected by scripts often arrives that way, and blocking it would corrupt what we are trying to measure. So a rendered page may show more lines in your log than the requests we initiate. Those extra lines are not counted against our budget, because we did not initiate them — but we do record how many there were, and can tell you the number if you ask.

↑ Contents

What it collects

  • Schema.org structured data (JSON-LD, Microdata, RDFa)
  • The presence and content of agent-facing files
  • HTTP response codes and headers
  • Platform signatures visible in page source

What we store. Extracted structured data fields, HTTP response headers, and short diagnostic fragments — the first 200 characters of files such as llms.txt, and up to 500 characters of a response body when a request is refused, so we can tell one kind of refusal from another. We do not store full page content or images.

Personal data. We study organisations, not individuals. Where a page marks up an author, we record only whether the markup type is Person — we never record the name itself. However, contact details published by an organisation — an email address, a phone number, a street address — are collected, and for sole traders these may also be personal data under UK GDPR. We process them on the basis of legitimate interest in research into public web infrastructure, we do not combine them with other sources, and we do not publish them. If you want your domain removed, see below.

↑ Contents

What we do with it

We publish aggregate findings — percentages, distributions, patterns across platforms. Individual companies are never named in a negative context. Raw data identifying specific domains is not published.

Aiprobase is a commercial company; we sell a product in this field. That is a conflict of interest and we state it in every publication that uses this data.

↑ Contents

How to stop it

Add this to your robots.txt:

robots.txt
User-agent: Aiprobase-Research
Disallow: /

We check robots.txt before every request and honour its Disallow rules.

We also honour Crawl-delay up to 30 seconds. If your robots.txt asks for more than that, we skip your domain entirely rather than crawl it faster than you asked.

We honour the X-Robots-Tag response header. If your homepage returns noindex or noai — either for all robots or addressed to aiprobase-research — we stop: nothing else is requested from your domain. By then we will have requested only robots.txt and the homepage — on each host your homepage redirected us through. If you want us not to request anything at all, use robots.txt or write to us.

A 401, 403 or 429 response is treated as a refusal and we do not retry or attempt to work around it.

If you would rather write to us: hello@aiprobase.com. Tell us the domain and we will exclude it from all future studies, no questions asked. We reply within 3 working days; the exclusion itself takes effect as soon as we read your message.

↑ Contents

How to verify it is really us

A user-agent string is just text and anyone can send one. To confirm a request came from us:

  • Check the source IP against our published range: https://www.aiprobase.com/research/crawler/ips.json. Our crawler currently runs from a single address, 145.241.222.58.
  • Or run a reverse DNS lookup on the source IP. Ours resolves to research.aiprobase.com, and research.aiprobase.com resolves back to the same IP. Check both directions: a reverse lookup alone can be faked by whoever controls the address.

If you see Aiprobase-Research from an address outside our published range, it is not us. We would like to know about it.

↑ Contents

Contact

AIPROBASE LTD, registered in England and Wales, Company No. 17129821. Registered with the UK Information Commissioner's Office, registration ZC192938.

Questions, complaints, exclusion requests: hello@aiprobase.com — we reply within 3 working days.

Last updated: 7 November 2026