How the robots work
Robots on the web are programs that read web pages by themselves. There are many kinds, and they do different things. Here you can see step by step what each kind does, and some of the robots we have met.
New here? How it all fits together
Search robots
A search robot reads web pages for a search engine, like Google or Bing. It wants to know what each page says, so the search engine can show the right page when you search for something.
What it does
- First it reads robots.txt, the note by the door that says which pages robots may read.
- Then it fetches a page and reads everything on it.
- The page has links to other pages. The robot follows them, one by one.
- What it has read it files in the search engine's index, like cards in a big card drawer.
- After a while it comes back to see what has changed.
Does it read robots.txt?
The big search robots say they do what robots.txt says. If a page is closed to them, they do not read it.
If you have a website
If you want people to find your site, let the search robots in. If there is something they should not read, write it in robots.txt. But something secret needs a password, because robots.txt is only a sign.
Sources, read 3 November 2026: developers.google.com (opens in a new window outside Pyssly) · rfc-editor.org (opens in a new window outside Pyssly)
AI robots
AI companies have several kinds of robots. Some collect text that an AI program can learn from. Others fetch pages when someone asks the company's chatbot about something.
What it does
- The robots that collect text read robots.txt first, the companies say.
- Then they fetch page after page and put the text in a big sack.
- From everything collected, the AI program learns how language works.
- Another day a person asks the chatbot about something.
- Then the company may send a fetcher, which fetches, for example, just that page.
- The fetcher brings the page back, and the chatbot answers with its help.
Does it read robots.txt?
The robots that collect text say they listen to robots.txt. Fetchers are different: when a person has asked for the page, some companies say robots.txt does not always apply.
If you have a website
In robots.txt you can say no to the robots that collect text for AI, each by its name, and still let the search robots in. The companies list their robots' names on their own pages.
Some robots of this kind
Sources, read 3 November 2026: developers.openai.com (opens in a new window outside Pyssly) · support.claude.com (opens in a new window outside Pyssly)
SEO tools
An SEO tool is a robot that measures other people's websites. Companies use it to see how their own site shows up in search engines, compared with others.
What it does
- It reads robots.txt first.
- Then it measures the page: what it is called, how big it is and how quickly it loads.
- It counts the links: which pages link here, and where this page links to.
- It all becomes a report with bar charts, which the company behind the tool sells to others.
Does it read robots.txt?
The big SEO tools say they follow robots.txt, and some go slower if you ask them to.
If you have a website
If you do not want to be measured, you can say no to them one by one in robots.txt. With search engines like Google and Bing it does not change whether your site shows up, but some of these tools also collect for a search engine of their own.
Some robots of this kind
Sources, read 3 November 2026: ahrefs.com (opens in a new window outside Pyssly) · semrush.com (opens in a new window outside Pyssly)
Data collectors
A data collector fetches lots and lots of web pages and keeps them as data. Sometimes that becomes an open archive anyone may use, sometimes a service sold to others.
What it does
- It reads robots.txt first.
- Then it fetches page after page, very many of them.
- It ties the pages together into one big bundle.
- Others then use the bundle, for example for research or to teach AI programs.
Does it read robots.txt?
The well-known data collectors say they do what robots.txt says.
If you have a website
If you say no to a data collector in robots.txt, your new pages do not go into its collections. What has already been collected may stay.
Some robots of this kind
Sources, read 3 November 2026: commoncrawl.org (opens in a new window outside Pyssly) · diffbot.com (opens in a new window outside Pyssly)
Link peekers
A link peeker makes the little cards with a picture and a heading that pop up when someone pastes a link into a chat or a post.
What it does
- Someone pastes a link into an app on their phone.
- The app's robot fetches the page the link leads to, right away.
- It picks out the page's heading, a sentence and a picture.
- From them it makes a little card that shows in the chat.
Does it read robots.txt?
Sometimes. Some link peekers follow robots.txt, others fetch the page anyway, because a person shared the link.
If you have a website
Put a clear heading, a short description and a picture in your page's code, and the cards will look nice. If you shut the link peekers out, the link is just a line of text.
Some robots of this kind
Sources, read 3 November 2026: developers.facebook.com (opens in a new window outside Pyssly)
Scripts
A script is a small program someone has written to fetch pages. It may be someone trying out their own site, or someone who wants to copy other people's.
What it does
- Someone writes a list of which pages to fetch.
- The program starts and fetches the first page.
- Then the next and the next, just as the list says, as fast as it can.
- When the list ends, it stops. It may never have said who it was.
Does it read robots.txt?
It depends. A script does what is on its list, and only reads robots.txt if whoever wrote it asked it to.
If you have a website
robots.txt does not help against a script that does not read it. What helps is locking up what should be secret and making whoever asks too often wait.
Some robots of this kind
Sources, read 3 November 2026: rfc-editor.org (opens in a new window outside Pyssly) · rfc-editor.org (opens in a new window outside Pyssly)
Domain watchers
Every day thousands of new web addresses are made. A domain watcher is a robot that keeps track of them and looks in on each new website.
What it does
- All the time, new addresses are registered around the world.
- The robot gets a list of the day's new addresses.
- It knocks on the front page of the new websites.
- What it sees it writes into a register, which is shown or sold to others.
Does it read robots.txt?
Sometimes. Some read robots.txt first, others do not.
If you have a website
They often come soon after a website is created. If you do not want them, you can say no to them in robots.txt, but the sign only stops the ones that listen.
Some robots of this kind
Sources, read 3 November 2026: czds.icann.org (opens in a new window outside Pyssly)
Burglars
A burglar is a robot looking for a way in. It goes from website to website trying doors many sites have, for example the login to WordPress.
What it does
- The robot puts on a mask and goes to a website.
- It knocks on doors that might be there, like /wp-login.php or /.env.
- It tries keys: common passwords, one after another.
- If the door is not there, or the lock holds, it moves on to the next website.
Does it read robots.txt?
Do not count on it. robots.txt is a sign, not a lock, and someone looking for a way in can walk past it.
If you have a website
Keep everything up to date, use long passwords and two-step sign-in, and never put secret files where the web can reach them.
Some robots of this kind
Sources, read 3 November 2026: developer.wordpress.org (opens in a new window outside Pyssly) · rfc-editor.org (opens in a new window outside Pyssly)
In disguise
A robot in disguise pretends to be an ordinary web browser, like the one you use yourself. It does not want the site to understand that it is a robot.
What it does
- The robot takes out a mask.
- With the mask on, it says: I am an ordinary web browser!
- But it can read page after page, much faster than a person could.
- And the sign by the door, robots.txt, it can walk past, because ordinary web browsers do not read it.
Does it read robots.txt?
Do not count on it. Ordinary web browsers do not read robots.txt, so something pretending to be one can skip it too.
If you have a website
A robot in disguise cannot be stopped with robots.txt. What helps is the same as against scripts: lock up what is secret and make whoever asks too often wait.
Some robots of this kind
Sources, read 3 November 2026: rfc-editor.org (opens in a new window outside Pyssly) · rfc-editor.org (opens in a new window outside Pyssly)
Unknown
An unknown robot says it is a robot but not which one, or says nothing at all. Then nobody knows who sent it, or why.
What it does
- It comes out of the fog.
- It only says: I am a robot. No name.
- It fetches a page, or a few.
- Then it disappears into the fog again.
Does it read robots.txt?
Sometimes. Some read robots.txt, but without a name the sign cannot say anything to them in particular, only to all robots at once.
If you have a website
A rule in robots.txt for all robots, User-agent: *, also applies to the unknown ones, if they listen.
Some robots of this kind
Sources, read 3 November 2026: rfc-editor.org (opens in a new window outside Pyssly) · rfc-editor.org (opens in a new window outside Pyssly)
You can learn what robots.txt is in Robot school, on the page The robot jar.