How to block search bots from accessing your site?

In order to block all bots from accessing your site, you should create a robots.txt file with the following content:

User-agent: *
Disallow: /

To block all bots from accessing a specific folder, you should create a robots.txt file with the following content:

User-agent: *
Disallow: /folder/

Please note that blocking all bots will get your site deindexed from search engines.

  • 0 Users Found This Useful
Was this answer helpful?

Related Articles

How do I disable directory browsing?

By default, if you don't have an index file in a web accessible folder, when accessed through a...

What do you mean when you say gigabyte, megabyte, GB, and MB?

The smallest unit of measurement on a computer is the bit. There are almost always eight bits in...

Can people find out where I live based on my domain registration information?

Unfortunately, the answer is usually going to be yes. Any address and contact information you...

Why does my website look different on different browsers?

Having a website look the same on different browsers has been an ongoing issue for the web...

Propagation Times

Propagation is the process of updating a domain with new nameserver (DNS) information across all...