ExiReachBot

What it fetches

ExiReachBot fetches public pages only. These are all of them:

• robots.txt, on every site, before any other page on it.
• Business websites: the home page, and up to six pages on the same site that look like contact or about pages. From them it reads the business contact details published there (such as an info@ address, a phone or a WhatsApp number), what the business says it does, links to its social profiles and app store listings, the tools the site is built with, and its logo and colours.
• For a business we prepare an app preview for: the logo image its home page points to, and, for a car or taxi business, up to two pages listing its vehicles or fares, with their pictures.
• For one of our clients, at their request: their own website, up to sixteen pages, and its sitemap.xml when its pages link to few others.
• Directories and list sites that might link to one of our clients: the home page and pages such as /contact, /about and /write-for-us, for a general address such as editor@ to write to.
• A single public page a user of our product asks it to read.
• Google Play developer and app listing pages, to see whether a business publishes an Android app.
• Links cited in articles written with our product, with one HEAD request each (or a one-byte GET where HEAD is not supported), to check that the page still exists. Links to social networks such as LinkedIn, Facebook and X are not checked.

Some pages are loaded in a headless browser, which also loads the images, styles and scripts the page itself asks for. Separately, we look apps up in Apple’s public App Store search API.

How it identifies itself

Every request it sends to a website carries one of these user agents:

ExiReachBot/1.0 (lead-enrichment; +https://exireach.com/bot)
ExiReachBot/1.0 (link-check; +https://exireach.com/bot)
ExiReachBot/1.0 (+https://exireach.com/bot)

What it does not do

• It does not attempt to log in, or access anything behind authentication.
• It does not submit forms or interact with your site.
• It does not collect personal information beyond published business contact details.
• It does not fetch a page your robots.txt disallows for it.

Blocking it

Add this to your robots.txt:

User-agent: ExiReachBot
Disallow: /

A group that names ExiReachBot is followed in place of the one for all crawlers, and the rules are read as RFC 9309 describes: Allow and Disallow, with * and $, the longest matching rule deciding. So Disallow: /contact under that line closes only your contact pages. We keep a copy of your robots.txt for up to ten minutes, so a change takes effect within ten minutes.

If your site has no robots.txt (it answers 404), we read it as allowing everything. If your server answers the request for robots.txt with an error, or does not answer, we read nothing on your site until it does.

Removing your details

If we hold contact details for your business and you would like them deleted, email [email protected]. We delete the record rather than flagging it, and we keep an irreversible hash of the address so that it is not collected again.