The purpose of the Robots TXT File is to dissuade bots from crawling certain website directories.
The default Joomla! 4 Robots TXT File includes the following line:
- Disallow: /api/
to dissuade bots from crawling a directory called API which was introduced with the launch of Joomla! 4.
Unfortunately, the Joomla! CMS update process does not overwrite the existing Robots TXT File.
It is therefore necessary to manually edit the file after updating to Joomla! 4.0 and later versions.
Adding the above line will serve to dissuade bots from crawling the new directory.
You can edit the file via the File Manager screen of your website's Hosting Control Panel.
In the cPanel Hosting Control Panel you will find a link to the File Manager in the panel labelled Files.
Read more: Hosting Control Panel.
Personal Joomla! CMS Help & Support
We coach, help and support managers with responsibility for Joomla! websites in organisations across Cheshire, Manchester, Merseyside, North West England & the UK.
Your personal data is nobody's business.