The purpose of the Robots TXT File is to dissuade (rather than prevent) automated search engine bots from crawling certain website directories.

Not all bots will respect this request for privacy.

For this reason you should password protect the administrator folder of your website and any other directory you do not want them to crawl.

The best way to do this is by using a Web Application Firewall.

Read more: Web Application Firewall.

 

Reasons to check your website's Robots TXT File

Contents include:

History of the Joomla! CMS Robots TXT File.
Reasons to check your website's Robots TXT File.
The robot.txt file and XML Site Maps.

Subscribers: sign in to read full article