Our web fetcher

What it is

The part of TreadLightlyAI that reads a web page so the assistant can answer a person's message.

When it runs

Only while answering someone's message: for a link they pasted, or for one of the assistant's own web search results when the search summary was not enough. It is not a crawler.

How to spot it

The user agent below, in the request headers.

robots.txt

Honored. It reads your robots.txt before it opens a page.

Contact

hello@treadlightly.ai

A person using TreadLightlyAI pastes a link into a conversation, or asks something the assistant answers by searching the web. When a search result's short summary does not hold the answer, the assistant may open that page on its own, without the person naming it. The fetcher then requests that one page so the assistant can answer from what it says and show where the answer came from.

Nothing else triggers it: it opens only a link the person pasted or a page from that message's search results. It is not a crawler: it does not follow links to discover pages, build an index, or visit on a schedule, and it reads at most two pages for any one message.

Each request carries this user agent:

TreadLightlyAI-WebFetch/0.1 (+https://treadlightly.ai/fetcher)

• A plain GET request for your robots.txt, then one for the page, on port 80 or 443.
• It asks for HTML or plain text. It sends no cookies and does not log in.
• It follows up to three redirects for a page (five for robots.txt) and stops reading a page after about two megabytes.
• It reads the text of the page and does not run scripts, so content that only appears after JavaScript runs is not seen.
• It does not fetch pages on private or internal network addresses.

Our fetcher reads your robots.txt before it opens a page and does not fetch a page your rules disallow. It follows the group for TreadLightlyAI-WebFetch if you have one, and otherwise the group for every user agent (*). It understands the usual wildcards, and the most specific matching rule wins. To turn it away from your whole site:

User-agent: TreadLightlyAI-WebFetch Disallow: /

• No robots.txt (a 4xx response) means every page is allowed.
• If your server returns an error or does not answer, it does not fetch the page.
• When a page redirects to another site, that site's rules apply too.
• It keeps a copy of your file for up to 24 hours, so a change can take a day to reach it.
• It ignores Crawl-delay, since it opens one page for one message rather than crawling.

If the fetcher is causing you a problem, tell us at the address below.

Write to hello@treadlightly.ai and include the URL, the time of the request, and the user agent from your logs if you have them.