PearFareBot
You are likely reading this because PearFareBot appeared in your server logs. This page explains who we are, what the crawler does, and how to control or block its access to your site. If you want us to stop, section 5 tells you two ways to do that, one of which we act on directly.
PearFareBot identifies itself with this user-agent by default:
PearFareBot/1.0 (+https://trypearfare.com/bot)01What PearFareBot is
PearFareBot is the crawler for PearFare's Managed Collection service. PearFare runs a batch data-extraction business: clients name the sources they need, and we fetch the publicly accessible pages from those sources so the content can be turned into structured data for them. PearFareBot is the component that does the fetching.
Every request the crawler makes is an ordinary outbound HTTP request from PearFare infrastructure, made at a client's instruction, at controlled request rates. It reads public web pages the same way a browser does. It does not sign in, and it does not carry any client's credentials. Our full commitments about this are on the security page, and the service is defined in our terms.
02How it behaves
| Behavior | What we do |
|---|---|
| Identification | Requests are made under the PearFareBot user-agent by default, with this page's URL in the string so you can find us. We do not disguise the crawler to get past a block. |
| robots.txt | We request and honor your robots.txt. Rules apply to the user-agent we present, and a User-agent: * rule always applies to us. See section 4. |
| Request rate | Conservative by default, one request per second per host or slower. If your robots.txt sets a Crawl-delay, we use the longer of the two. |
| Scope | We fetch a bounded set of pages a client names, within a single client-named domain. We do not crawl the wider web from your site. |
| Public pages only | We collect only pages that are publicly accessible without a login, payment, or human-verification challenge. Anything else is skipped and recorded, never bypassed. |
| Backing off | On a 403, 429, or a challenge page, we stop fetching that source rather than retrying around it. |
| Audit trail | Every URL we touch is logged with its outcome, so a source can be shown exactly what was and was not collected. |
03What it will not do
These limits are fixed. They are not settings a client or an operator can turn off, because they are commitments in our terms and on our security page.
- It does not send credentials, cookies, or tokens, and it does not authenticate to your site.
- It does not collect anything behind a login, a paywall, or a human-verification challenge, and it does not attempt to solve one.
- It does not bypass any access control. A page that is not publicly accessible is not collected.
- It does not rotate addresses or spoof its identity to evade a block you have put in place.
04Block or slow it in robots.txt
PearFareBot obeys the Robots Exclusion Protocol. Place a robots.txt file at the root of your site and we will follow it on the next visit. A few common rules:
Block PearFareBot from your whole site:
User-agent: PearFareBot
Disallow: /Block one section only:
User-agent: PearFareBot
Disallow: /members/
Disallow: /checkout/Allow it, but slow it down:
User-agent: PearFareBot
Crawl-delay: 10A User-agent: * rule that disallows crawling always applies to us as well, so if you already block all bots, you already block PearFareBot. Targeting PearFareBot by name lets you set a rule for us without changing how other crawlers see your site.
05Ask us to stop
If you would rather not manage a robots.txt rule, or you want to be certain, email us and we will add your site to a permanent do-not-collect list. That list is honored regardless of any client instruction.
- Email [email protected] with your domain in the subject line.
- Tell us which site or paths you want us to leave alone, or say "all of it."
- We confirm and stop. There is nothing you need to prove and no account required.
If you believe PearFareBot has fetched something it should not have, tell us at the same address and we will look into it and correct our behavior.