Search Engine Friendly Robots TXT File
The purpose of the Robots TXT File is to dissuade (rather than prevent) automated search engine bots from crawling certain website directories.
Not all bots will respect this request for privacy.
For this reason you should password protect the administrator folder of your website and any other directory you do not want them to crawl.
The best way to do this is by using a Web Application Firewall.
Read more: Web Application Firewall.
Reasons to check your website's Robots TXT File
To improve performance in search engines
Prior to the release of Joomla! 3.3 the default Robots TXT File included with the Joomla! CMS contained these three lines:
- Disallow: /images/
- Disallow: /media/
- Disallow: /templates/
resulting in search bots being dissuaded from crawling website media content, and from verifying its mobile-friendliness.
Because the Joomla! CMS update process does not overwrite the existing Robots TXT File it is necessary to manually edit the file in later Joomla! versions if you want to remove these three lines to increase your website's visibility in search results.
You can edit the file via the File Manager screen of your website's Hosting Control Panel.
To dissuade bots from crawling the API directory after updating to Joomla! 4
Following the launch of the Joomla! 4 series, a new line was added to the default Robots TXT File:
- Disallow: /api/
to dissuade bots from crawling a new directory called API which was introduced with Joomla! 4.
Because the Joomla! CMS update process does not overwrite the existing Robots TXT File it is necessary to manually edit the file in Joomla! 4.0 and later versions if you want to dissuade bots from crawling this new directory.
You can edit the file via the File Manager screen of your website's Hosting Control Panel.
