Build a clean, standards-compliant robots.txt file in a few clicks, with presets for popular platforms.
Place this file at the root of your domain: https://yoursite.com/robots.txt
A robots.txt file tells search-engine crawlers which parts of your site they may crawl. Configured well, it preserves crawl budget and protects sensitive areas. Configured badly, it can accidentally block important pages from being indexed — so it pays to check it carefully.
Warning: blocking a page in robots.txt does not always keep it out of Google. To de-index a page, use a meta noindex tag instead.
At the root of your domain, reachable at https://yoursite.com/robots.txt. It will not work inside a subfolder.
You can disallow technical areas (/admin/, /cgi-bin/), but do not rely on it for security — the file is public.
Yes, it is recommended. It helps search engines discover all of your important URLs faster.
