SEO optimization and robots.txt file

The robots.txt is a file containing information about which folders should be crawled and which should not by search engine bots.
Since 2014, google has announced that "Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings."

The default robots.txt file provided by joomla assumes that all CSS and js files are in the media folder, which is not the case of a lot of Joomla extensions (modules, components, plugins) and in special for Virtuemart extensions.

To avoid to be penalized by google, we recommend you to modifiy the default robots.txt file.

The robots.txt file is on the joomla root. The default content of this file since Joomla 3.3 is:

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

All folders not mentionned in this file are Allow.

In the case of VirtueMart,  add the following lines to the file robots.txt:

Allow: /components/com_virtuemart/assets/
Allow: /modules/mod_virtuemart_cart/assets/

If you want to be sure, what to change, please use the Webmaster Tools of the most common Search Engines.