When convenient

Search engines are allowed to index everything on your site

What it means

robots.txt is the plain-text file at https://example.com/robots.txt that tells search engines which parts of your site to skip. Yours exists, but no Disallow: line in it names a path. Whether it says Allow: / or has an empty Disallow:, the meaning is the same: everything is allowed.

That is a valid file, and for a very simple site it can be fine. But it means every page a crawler can find is open for listing, including pages that exist only for you or your logged-in customers.

This often happens when a default file from a platform or plugin is replaced with a minimal one, or when an "allow everything" example is copied in while fixing a different problem.

Why it matters

Admin logins, account pages and internal search results can end up in search results. That makes your login pages easy to find for anyone searching for sites to attack, and it clutters your listings with pages customers do not need.

Internal search result pages are a particular waste. Every search creates a new address, and crawlers can spend their time on those instead of your real pages.

Adding rules does not make those pages secure. robots.txt is a public, polite request that search engines follow and attackers ignore. It reduces exposure. Passwords and two-factor login do the actual protecting.

How to check your own site

1. Open https://yourdomain.com/robots.txt in a browser. Look for lines that start with Disallow: followed by a path, such as Disallow: /wp-admin/. If there are none, this alert applies.
2. List the private areas your site actually has. Common ones are an admin or dashboard area, a login page, customer account pages, the cart and checkout, and search results. Visit each one and note the path after your domain, such as /account/.
3. Search Google for site:yourdomain.com login or site:yourdomain.com account to see if any of those pages are already listed.

How to fix it

Any website

1. Open your robots.txt in a plain text editor. Keep the User-agent: * line, and keep your Sitemap line if you have one.
2. Add one Disallow: line for each private area, using the paths you noted. Each path starts with a slash. A path ending in a slash, like /account/, covers everything inside that folder.
3. You can remove Allow: /. Anything not disallowed is allowed anyway, so the line adds nothing.
4. Never add Disallow: / on its own. That asks search engines to skip your whole site.
5. Upload the file back to the same place, then reload it in a browser to confirm the change.

User-agent: *
Disallow: /admin/
Disallow: /login/
Disallow: /account/
Disallow: /checkout/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

WordPress

1. The standard WordPress rules block the admin area but leave open the one admin file that the public side of the site uses. Start with the lines below, and add any other private paths your site has, such as a members or account area.
2. If an SEO plugin manages your robots.txt, edit it there, usually under the plugin's Tools or settings page, so your change is not overwritten.
3. Do not block /wp-content/ or /wp-includes/. Those folders hold the images, styles and scripts search engines need to see your pages properly.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/wp-sitemap.xml

Shopify, Squarespace, Wix and other hosted builders

1. Shopify and Squarespace ship with default rules that already block carts, checkouts, accounts and admin pages. If yours blocks nothing, someone probably customized it.
2. On Shopify, look in your theme code for a robots.txt.liquid template, and remove the customization or restore the default rules. Squarespace does not allow edits, so if you see this on Squarespace, check that your domain really points to your Squarespace site.
3. Wix has a robots.txt editor in its SEO settings. Add Disallow lines there for any member or account pages your site has.

cPanel hosting

1. Open "File Manager", go to public_html, right-click robots.txt and choose Edit.
2. Add your Disallow lines, save, and check the file in a browser.

What to send your developer

robots.txt returned 200 with an Allow: / rule and no Disallow rule that names a path. Add Disallow rules for admin, login, account, cart and checkout, and internal search paths, for example /wp-admin/, /admin/, /login, /account/ and /search. Rules are case-sensitive prefix matches grouped under User-agent lines, and Google also supports * and $ wildcards. Disallow stops crawling, not indexing: a blocked URL can still be listed without a snippet if other pages link to it, and a noindex tag on a blocked page is never seen. For pages that must stay out of the index, allow crawling and send X-Robots-Tag: noindex. Do not list paths you want kept secret, since the file is public.

Common questions