# AI crawlers explained | Propel

Source: https://runpropel.com/glossary/ai-crawlers

Glossary

# What is ai crawlers?

AI crawlers are the programs AI companies send to read web pages. Some collect pages to train models, some build a search index for an assistant, and some fetch a page live because a person asked about it. Each one names itself in your server logs.

## What kinds are there?

- Training crawlers gather text for future models, for example GPTBot and ClaudeBot.
- Search crawlers build the index an assistant searches, for example OAI-SearchBot, Claude-SearchBot and PerplexityBot.
- User fetchers open one page because someone asked, for example ChatGPT-User, Claude-User and Perplexity-User.

Your robots.txt can allow or block each by name.

## Can a crawler be faked?

Yes. Anyone can put "GPTBot" in a request. The companies publish the addresses their crawlers use, so a visit can be checked against those before it is counted.

## What does Propel show?

From your hosting logs, Propel can show which AI crawlers read your site and which pages they read, with fakes filtered out by name and address. See [Get found](https://runpropel.com/get-found#search).

Where Propel does this

[Get found →](https://runpropel.com/get-found#search)

Related terms

- [llms.txt](https://runpropel.com/glossary/llms-txt)
- [Answer engine optimisation (AEO)](https://runpropel.com/glossary/answer-engine-optimisation)
- [AI visibility](https://runpropel.com/glossary/ai-visibility)
- [All terms](https://runpropel.com/glossary)