The purpose of the Robots TXT File is to dissuade automated search engine bots from crawling certain website directories.
In old versions of Joomla! CMS, the Robots TXT File dissuaded bots from crawling these three directories:
by inserting three lines of text in the default Robots TXT File:
- Disallow: /images/
- Disallow: /media/
- Disallow: /templates/
resulting in bots not crawling a website's media content and verifying its mobile-friendliness.
Unfortunately, the Joomla! CMS update process does not overwrite the existing Robots TXT File.
So if your website was updated from an older version of Joomla! you should check its Robots TXT File.
You can do this via the File Manager screen of your website's Hosting Control Panel.
If you see the above three lines, remove them to improve your website's visibility in search results.
Personal Joomla! CMS Help & Support
We coach, help and support managers with responsibility for Joomla! websites in organisations across Cheshire, Manchester, Merseyside, North West England & the UK.
Whilst Wynchcote Limited can offer advice for how to improve your search engine ranking, we do not offer any guarantees that your rankings will improve as a result of this advice.