SEO glossary

    What is Robots.txt?

    Robots.txt is a file at the root of a website that tells crawlers which areas they may or may not visit.

    Why it matters

    A wrong rule can block search engines or AI crawlers from your whole site.

    Related terms

    XML Sitemap

    An XML sitemap is a file listing the pages you want search engines to find and i…

    llms.txt

    llms.txt is a plain-text file at a site's root that summarises its most importan…

    Crawl Budget

    Crawl budget is the number of pages a search engine will crawl on a site within …

    Let Rank Pilot fix it for you

    Free website analysis. Automated fixes from £29/mo.

    Analyse my website freeStart free