The purpose of the Robots TXT File is to dissuade bots from crawling certain website directories.
The default Joomla! 4 Robots TXT File includes the following line:
- Disallow: /api/
to dissuade bots from crawling a directory called API which was introduced with the launch of Joomla! 4.
Unfortunately, the Joomla! CMS update process does not overwrite the existing Robots TXT File.
It is therefore necessary to manually edit the file after updating to Joomla! 4.0 and later versions.
Adding the above line will serve to dissuade bots from crawling the new directory.
You can edit the file via the File Manager screen of your website's Hosting Control Panel.
In the cPanel Hosting Control Panel you will find a link to the File Manager in the panel labelled Files.
Read more: Hosting Control Panel.
JOOMLERS.UK is sponsored by WYNCHCO Solutions
WYNCHCO Solutions help and support managers responsible for Joomla! CMS websites in UK business and third sector organisations across Cheshire, Greater Manchester, Merseyside and North West England.
