{"id":227,"date":"2012-05-15T07:22:13","date_gmt":"2012-05-15T07:22:13","guid":{"rendered":"http:\/\/www.redesigns.org\/web-builder\/blog\/?p=227"},"modified":"2025-06-03T18:57:44","modified_gmt":"2025-06-04T01:57:44","slug":"how-robots-can-keep-your-site-from-being-indexed-on-google-yahoo-top-search-engines","status":"publish","type":"post","link":"https:\/\/www.ultimatewb.com\/blog\/227\/how-robots-can-keep-your-site-from-being-indexed-on-google-yahoo-top-search-engines\/","title":{"rendered":"How robots can keep your site from being indexed on Google, Yahoo, top search engines"},"content":{"rendered":"\n
\n \"robot-preventing-website-from-being-crawled-and-indexed\"\n\n <\/picture>\n <\/a><\/figure>\n\n\n\n

Yes, robots can actually prevent your website from being indexed by top search engines like Google and Yahoo – and no, this isn’t a sci-fi movie plot! Don’t worry, you can fix it, there is an easy solution.  …Just to clarify first though, we are talking about robots – as in a robotx.txt file – in the root of your website, i.e. www.your-website.com\/robots.txt .<\/p>\n\n\n\n

The Quick Fix: Use a Proper robots.txt File, Don\u2019t Delete It<\/strong><\/h2>\n\n\n\n

If you want your website to be indexed – all<\/em> of your webpages – do not delete the robots.txt<\/code> file.<\/strong><\/p>\n\n\n\n

In the past, removing it might have helped, but today, tools like Google PageSpeed Insights<\/a><\/strong> expect a properly configured robots.txt<\/code> file.
Having none at all can trigger a warning and even lower your performance score<\/strong>.<\/p>\n\n\n\n

That\u2019s why all versions of Ultimate Web Builder software<\/strong> come with a default robots.txt<\/code> file that\u2019s designed to work correctly – allowing indexing and keeping SEO tools happy.<\/p>\n\n\n\n

How to Keep Your Entire Website Open to Search Engines<\/strong><\/h2>\n\n\n\n

To make sure your entire website is crawlable and indexable, your robots.txt<\/code> file should explicitly allow everything. A simple and effective file would be:<\/p>\n\n\n\n

User-agent: *
Disallow:
<\/code><\/pre>\n\n\n\n
This tells all search engine bots<\/strong> (*<\/code>) that nothing is off-limits<\/strong>, so your whole site can be indexed. This setup keeps your site fully accessible to search engines and satisfies performance tools like PageSpeed Insights.<\/p>\n\n\n\n
Blocking Specific Crawlers (Optional)<\/strong><\/h2>\n\n\n\n
You might come across a more advanced setup like this:<\/p>\n\n\n\n
User-agent: ia_archiver
Disallow: \/

User-agent: *
Crawl-delay: 120
<\/code><\/pre>\n\n\n\n
Here\u2019s what it does:<\/p>\n\n\n\n