by (1.4k points)
Hey All.
How should a professional robots.txt for seotoaster look like?
Which directories should be blocked?
/cache/ ??

Best regards

Ronny

1 Answer

by (4.9k points)
Hello Ciatronical

For example you should use:

User-agent: *
Disallow: /system/
allow: /system/css
allow: /system/js
Disallow: /tmp/
allow: /tmp/*.js
allow: /tmp/*.css
Sitemap: https://_valid_domain.com_/sitemapindex.xml
User-agent: googlebot
Allow: /themes/
Allow: /system/js
Allow: /system/css

Regards, Pavel.
by (1.4k points)
Thanks very much.

Best regards,
Ronny
...