Amp + Joomla's robots.txt

[Forsh]

The AMP crawler see my pages but the error for validating says:

Blocked by robots.txt
Status: Error

The attachment is from Google Webmaster Tools. Oddly, when I fetch with Googlebot everything shows as fine and the actual AMP pages from Joomla are fine. This is the AMP page indexed by the AMP project, you can see in the URL.

https://www-denverdonate-com.cdn.amppro ... ocrisy/amp

I'm not understanding what's happening that the AMP crawler can crawl, Googlebot fetch and render does fetch and render when checked in GWT....but that the robots.txt is causing them to have errors.

If you disallow the /amp/ directory if would do that but I'm not disallowing that directory and you can see that the Joomla pages are saved by AMPProject and render. It's not in the robots.txt at all.

User-agent: *
Disallow: /administrator/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /tmp/

Is there a setting inside the CMS itself that I'm missing or something?

denverdonateampblocked.jpg

It seems straightforward and the only crawler I list is MSN/Bing and all I do is change the crawl not even block entirely.

User-agent: bingbot
Crawl-delay: 1

User-agent: msnbot
Crawl-delay: 1

User-agent: msnbot-media
Crawl-delay: 1

[mjparadac]

Hello Forsh

In some cases, Google may not be able to reach your server. If the warning continues to appear, you can reach out to Google https://support.google.com/webmasters/a ... 9981?hl=en for further assistance.

Also you can check the sintax at http://www.robotstxt.org/orig.html