ExcuseThisReadiness

How to recognise it

Every request it makes sends this user agent:

ExcuseThisReadiness/1.0 (+https://excusethis.com/bot)

Requests come from Vercel's hosting network, which has no fixed list of addresses, so the user agent is the way to spot it.

What it fetches

For each check it asks for, in this order:

• /robots.txt, before anything else.
• The homepage, to confirm where the store lives. It follows a redirect between www and the bare domain, from http to https, or to another address on the same company domain (kowtowclothing.com to nz.kowtowclothing.com, say), but not a redirect to a different domain.
• The sitemaps listed in robots.txt, plus /sitemap.xml, to find product pages. From a large sitemap index it reads only a few of the product sitemaps.
• Up to 10 product pages.
• /.well-known/ucp and /llms.txt, if time and requests are left.

Every request is a plain GET. It doesn't run JavaScript, load images or stylesheets, fill in forms, log in or add anything to a cart. It doesn't follow links or redirects to other domains, except when robots.txt itself redirects to another domain's robots.txt.

Limits

Each check stays within these limits:

• 30 requests in total, with no more than 12 of them spent finding product pages.
• No more than 2 requests at a time.
• 8 seconds per request and 60 seconds for the whole check.
• 2 MB per response and 20 MB per check.
• Up to 3 redirects per request.

A complete result is kept for 24 hours, so asking about the same store again that day usually sends no requests at all. Any one store can be checked at most 12 times an hour, whoever asks, and a forced recheck is limited to twice an hour.

robots.txt

It reads robots.txt first and follows the group for ExcuseThisReadiness, or the * group if there isn't one. Pages it isn't allowed to fetch are skipped and shown in the report as not checked.

If robots.txt can't be read because of a server error, rate limiting, a timeout or a challenge page, the check stops there and nothing else is fetched. A robots.txt that returns 404, or another 4xx error apart from 429, counts as having no restrictions, as RFC 9309 sets out.

Blocking it

To keep it off your whole site, add this to your robots.txt:

User-agent: ExcuseThisReadiness
Disallow: /

A check of your site will then read robots.txt and stop. To keep it out of part of the site, disallow just those paths, for example Disallow: /account/, and it will skip them.

Questions

If it has caused a problem for your site, or you have a question about what it does, get in touch through the contact page.