# robots.txt explained | Propel

Source: https://runpropel.com/glossary/robots-txt

Glossary

# What is robots.txt?

robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may read. Each crawler is named by its user agent, so a site can let a search crawler in while keeping a training crawler out.

## How does it work?

The file lists groups of rules. Each group names one or more crawlers and the paths they may not read. A crawler that follows the standard reads the file first and skips what it is told to skip. The standard is [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).

## What does it mean for AI answers?

Blocking an AI company's search crawler can keep your pages out of the index its assistant searches, so the assistant can't quote them. Many sites block every AI crawler by accident, with one broad rule. robots.txt is a request, not a lock: well-behaved crawlers follow it, and others may not.

## What does Propel check?

Propel reads your robots.txt and says, crawler by crawler, which AI crawlers it lets in and which it keeps out, so a rule you didn't mean shows up. See [Get found](https://runpropel.com/get-found).

Where Propel does this

[Get found →](https://runpropel.com/get-found)

Related terms

- [AI crawlers](https://runpropel.com/glossary/ai-crawlers)
- [llms.txt](https://runpropel.com/glossary/llms-txt)
- [Citations in AI answers](https://runpropel.com/glossary/ai-citations)
- [All terms](https://runpropel.com/glossary)