The robot jar
New here? How it all fits together
Here we keep the robots that went in although the sign said no. The deeper they have gone, the lower in the jar they end up.
The lines in the jar are rooms in the maze: room 1, 10, 100 and 1,000. The further in a robot has gone, the lower it stands.
- –in the last seven days
- –kinds we have caught
- –deepest today
No robot to follow right now.
Follow a robot while you watch.
Robot school
What is a robot on the web?
A program that reads web pages by itself. A crawler goes from link to link with no person clicking, a bit like you leafing through a book, but lightning fast.
Source: developers.google.com, read 2 November 2026 (opens in a new window outside Pyssly)
What is robots.txt?
A small file a website can have, like a sign by the door. It says which rooms robots may go into. The sign cannot lock any door: it only helps if the robot reads it and listens.
Source: rfc-editor.org, read 2 November 2026 (opens in a new window outside Pyssly)
Search robot or AI robot?
A search robot reads pages so that a search engine can find them for you. An AI robot gathers text an AI program can learn from. And then there are fetchers: they don't wander about by themselves, but fetch a single page when a person asks an AI service for it. A company may say robots.txt does not apply to such a fetch, since a person asked.
Source: developers.openai.com, read 2 November 2026 (opens in a new window outside Pyssly)
Why don't they all listen?
Some robots never read the sign. Others read it but do as they like. And anyone can call their program anything. So we only know what a robot calls itself, not for sure who sent it.
Source: rfc-editor.org, read 2 November 2026 (opens in a new window outside Pyssly)
New here? How it all fits together
All the robots and their cards are in the craft cupboard.