About SitemapDigitalBot

If you found this page from a line in your server logs, this explains what visited you and how to stop it.

What it is

SitemapDigitalBot is the crawler behind sitemap.digital, a free tool that maps a site structure and reports orphan pages, click depth and broken internal links. It runs only when somebody pastes a URL into the site. There is no background or scheduled crawling, and no crawl happens unless a person asked for one.

It identifies itself as:

SitemapDigitalBot/1.0 (+https://sitemap.digital/bot)

What it does on your site

• Reads robots.txt first and obeys it, including rules aimed at *.
• Reads your sitemap where you have one, and follows links from the homepage where you do not.
• Fetches a maximum of 20 pages for an anonymous scan, or 50 when someone has entered an email.
• Requests in small batches, a few seconds apart, rather than all at once.
• Reads raw HTML only. It runs no JavaScript, submits no forms, and stores no personal data from your pages.
• Sends no requests at all once the scan finishes.

How to block it

Add this to your robots.txt and the crawler will stop, on its next scan and every one after:

User-agent: SitemapDigitalBot
Disallow: /

We do not attempt to work around blocks, and we do not disguise the crawler as a browser. If a site returns 403 to us we report that honestly as blocked rather than guessing at what was behind it.

Getting in touch

If the crawler behaved badly on your site, or you want a scan of your domain deleted, email hello@sunnypatel.co.uk and it will be removed. Scans expire on their own after 90 days.