GallopingDigital-Crawler

Operated by Galloping Digital for the race-operations platform it runs for motorsport teams.

What it does

It reads public event information published by race organisers and sanctioning bodies (event schedules, supplementary regulations, bulletins, entry lists and results) so racing teams have the current documents for the events they attend.

How it behaves

• It identifies itself with the user agent GallopingDigital-Crawler/1.0 (+https://crawler.gallopingdigital.com/crawler; support@gallopingdigital.com). When it uses a browser to read a page, it appends the same identity to the browser's own user agent.
• It signs its requests with Web Bot Auth (HTTP Message Signatures). Its public key is published at /.well-known/http-message-signatures-directory, and every signed request carries Signature-Agent: "https://crawler.gallopingdigital.com".
• It honours robots.txt and Crawl-delay, reads only public pages, and requests at most about one page per second from any site, usually far less: it rechecks pages on a schedule tied to upcoming events and uses conditional requests so unchanged pages cost almost nothing.
• It never logs in, never solves CAPTCHAs and never disguises itself. If a site's protection refuses it, it stops and a person obtains the document instead.

Contact or opt out

To ask a question, report a problem or have your site excluded, email support@gallopingdigital.com. You can also block GallopingDigital-Crawler in your robots.txt; it will stop.