When convenient

Your site has no instructions file for search engines

What it means

robots.txt is a small plain-text file at the root of your site, at https://example.com/robots.txt. Search engines and other well-behaved crawlers read it before anything else to see which parts of the site you would like them to skip.

EverThread asked for that address and got a "not found" response, or no response at all. That means crawlers have no instructions from you, so they treat every page they can find as open.

The file only works at the root of your domain. A robots.txt inside a subfolder, such as /blog/robots.txt, is ignored.

Why it matters

Without it, search engines may crawl and list pages you would rather they skip, such as admin logins, account pages, shopping carts and internal search results. Those clutter your search listings and make your login pages easy to find.

It is also where you tell search engines where your sitemap lives, which helps them find new pages sooner.

Keep expectations realistic. robots.txt is a polite request, not a lock. Search engines follow it, but anyone can read the file and bad actors ignore it. Protect private pages with a password, not with robots.txt.

How to check your own site

1. Open https://yourdomain.com/robots.txt in a browser, using your own domain. If you see a "page not found" message or your site's 404 page, you do not have one.
2. If your site uses www, try that version too. Each hostname has its own robots.txt.
3. If you use Google Search Console, its robots.txt report, under Settings, shows what Google last fetched and when.

How to fix it

Any website

1. Create a plain-text file named exactly robots.txt, all lowercase. Use a plain text editor, not a word processor.
2. Start with the example below. Change the Disallow lines to paths that actually exist on your site, and change the Sitemap line to your real sitemap address.
3. Upload it to the top folder of your site, the same folder as your home page file. On many hosts that folder is called public_html or www.
4. Open https://yourdomain.com/robots.txt to confirm it shows your text.

User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

WordPress

1. WordPress normally creates a robots.txt on its own, so a missing one usually means something is in the way. Common causes are WordPress installed in a subfolder rather than at the root, or a caching, security or server rule that answers the request before WordPress sees it.
2. The easiest fix is an SEO plugin. Most include a robots.txt editor, usually under a Tools or settings page, that creates a real file for you.
3. Or upload your own file to the top folder of your site, which for most sites is the folder that contains wp-config.php. A real file takes priority over the one WordPress generates.
4. A good starting point for WordPress is below. Change the Sitemap line to your real sitemap address.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

Shopify, Squarespace, Wix and other hosted builders

1. These platforms create a robots.txt for you automatically, so this alert is unusual on them. Check that your domain is connected to the platform and that the address EverThread checks is the one the platform serves.
2. Shopify lets you customize the file by adding a robots.txt.liquid template in your theme code. Wix has a robots.txt editor in its SEO settings. Squarespace does not let you edit the file.

cPanel hosting

1. Open "File Manager" and go to public_html, or to the folder for your domain if it is an addon domain.
2. Click "+ File", name it robots.txt, then right-click it and choose Edit. Paste in your rules and save.
3. Open the address in a browser to confirm it works.

Apache or Nginx

1. Place robots.txt in the document root of each site.
2. If it still returns 404, look for a rule that sends every request to your application, or a location block that blocks .txt files, and add an exception for /robots.txt.

What to send your developer

GET /robots.txt returned 404 or failed. Serve a UTF-8 plain-text file with a 200 status at the root of each hostname and protocol. Google treats a 404 as permission to crawl everything, but a server error on robots.txt can make it slow down or pause crawling, so make sure the file never errors. Include an absolute Sitemap: URL and Disallow rules for admin, account and internal search paths. Do not block CSS or JavaScript, which search engines need to render pages. For pages that must stay out of search results, use a noindex meta tag or X-Robots-Tag header rather than robots.txt. See Google's robots.txt guide.

Common questions