The sign
What do the robots do at our robots.txt?
Robots.txt is a sign that shows where we ask robots not to go. Here you see what we have seen: robots that fetched the sign, robots that went into the maze and robots that knocked on our pretend doors. These are requests we counted, not proof of who followed the sign.
Days are counted in Swedish time. We show the name the robot gives itself, or what kind of visit it looked like. Several robots can use the same name, so one name can be in several lists.
At the sign
| Name or kind | Fetched the sign | Turned back at the door |
|---|---|---|
| 609 | 0 | |
| 98 | 0 | |
| 40 | 4 | |
| 21 | 0 | |
| 20 | 0 | |
| 15 | 1 | |
| – | 13 | |
| 9 | 0 | |
| 6 | 1 | |
| 5 | 0 | |
| 2 | 0 | |
| 1 | 1 | |
| – | 1 | |
| 1 | 0 | |
| All | 827 | 21 |
– counted only for robots that say they are programs.
Turned back = came to the door and did not go into the first room in the next ten minutes.
In the maze
| Name or kind | Went in | of which Soon after the sign | Rooms | Deepest |
|---|---|---|---|---|
| 211 | 3 | 211 | 1 | |
| 56 | 55 | 56 | 1 | |
| 17 | 17 | 17 | 1 | |
| 1 | 0 | 12 | 8 | |
| 11 | 11 | 11 | 1 | |
| 9 | 9 | 9 | 1 | |
| 8 | 0 | 8 | 1 | |
| 7 | 7 | 7 | 1 | |
| 6 | 6 | 6 | 1 | |
| 6 | 6 | 6 | 1 | |
| 6 | 6 | 6 | 1 | |
| 6 | 6 | 6 | 1 | |
| 6 | 6 | 6 | 1 | |
| 6 | 6 | 6 | 1 | |
| 4 | 4 | 4 | 1 | |
| 4 | 4 | 4 | 1 | |
| 3 | 3 | 3 | 1 | |
| 3 | 3 | 3 | 1 | |
| 3 | 3 | 3 | 1 | |
| 3 | 3 | 3 | 1 | |
| All | 376 | 158 | 387 | 8 |
Soon after the sign: We saw the sign fetched from the same network address at most an hour before. It need not be the same robot, and not every such fetch can be linked.
At the pretend doors
| Name or kind | Times |
|---|---|
| 27,238 | |
| 171 | |
| 169 | |
| 62 | |
| 52 | |
| 29 | |
| 26 | |
| 18 | |
| 17 | |
| 16 | |
| 16 | |
| 15 | |
| 14 | |
| 14 | |
| 13 | |
| 12 | |
| 12 | |
| 11 | |
| 11 | |
| 10 | |
| All | 27,926 |
The brake stopped 973 requests that came too close together.
Updated 16:53
What the robots give away
Mostly the last seven days, from what they sent and how they behaved. Click a robot's name to see its page.
- 21 times a robot came to the door and did not go into the first room in the next ten minutes: “Script dressed as a browser”, “Unknown robot” and “Like a real browser”.
- 27,926 times robots knocked on our pretend doors, which look like configuration files and WordPress logins. Most attempts came from “Burglar”, “Unknown robot” and “Script”.
- Not confirmed: “ClaudeBot”, “meta-externalagent” and “CCBot”.
- 158 of 376 walks began with the robot reading robots.txt, and then it went in anyway.
- 211 pages were fetched by “Script dressed as a browser”.
- Run scripts: “Like a real browser”.
- “Burglar” was asked to wait but asked again anyway, 825 times.
- “Script dressed as a browser” searched straight down, room after room.
- Most found their way here via: Other pages.
- Most came from these networks: Google, Amazon and Microsoft.
- Most came from: United States, Japan and Canada.
Which files does the thief want to steal? The thief's list







