About Worrl

Worrl is a search engine. It searches web pages, news, images and videos, and shows the newest of each without a search.

The Worrl crawler

To build its index Worrl reads public web pages with an automated program, a web crawler. The rest of this page explains how to recognise it, how it behaves, and how to control or stop it.

How to recognise it

The crawler identifies itself in the User-Agent header of every request:

WorrlCrawler/0.1 (+https://worrl.com/crawler)

It currently crawls from 13.63.227.254. This address may change; if you allow-list by address, write to us and we will tell you before it does.

When someone views image results, Worrl fetches a small preview of the image on their behalf. Those requests identify themselves as WorrlImageProxy/0.1 (+https://worrl.com/crawler).

What it does

• Fetches public pages only. It does not sign in, fill in forms, or run JavaScript.
• Stays on the site it was asked to crawl. It records which other sites a page links to (the site names only), not the pages behind them.
• Reads text, titles, descriptions, dates, and the images and videos a page points to.
• Removes pages from the index when they are gone (a 410 or repeated 404 response).

How politely it behaves

• It reads robots.txt before crawling a site and follows it, including Crawl-delay. If robots.txt cannot be fetched because of a server error, it does not crawl that site.
• It makes about one request per second to a site. If a site answers 429 or 503 it slows down, obeys Retry-After, and speeds up again only after a run of clean responses.
• It asks for unchanged pages cheaply, using ETag and Last-Modified, so a page that has not changed costs a short reply.
• It revisits busy news sites every few hours and most other sites daily to weekly, and visits quiet sites less often over time.

How to control it

To keep the crawler out of your whole site, add this to robots.txt:

User-agent: WorrlCrawler
Disallow: /

To keep it out of part of your site, or to slow it down:

User-agent: WorrlCrawler
Disallow: /private/
Crawl-delay: 10

To keep a single page out of the index, use <meta name="robots" content="noindex"> or an X-Robots-Tag: noindex header. Use nofollow to stop it following the links on a page. Changes to robots.txt take effect within a day.

Image previews are fetched for a visitor and do not read robots.txt. If you want your images kept out of Worrl, or a page removed from results, write to us and we will do it.

Contact

If the crawler is causing you a problem, or you want it to crawl your site, write to support@worrl.com. Please include your site's address and, for a problem, roughly when you saw it.