Joomla! Forum - community, help and support

Hi all,
I have recently started using screamingfrog SEO spider to crawl my site to help me find errors.
The spider is returning a large number of 403 forbidden access.
The pages in question load ok but when I examine my site with google site:www.aegeanescapes.com I notice that old meta descriptions are being used which makes me think that the errors are correct and google is not crawling the pages in question.
Can any one help by explaining how to remove this problem so that the pages can be crawled.
Many thanks

I have the same problem - were you able to resolve this issue?

Hi Desmotech,

Unfortunatly I have not found a solution at the moment

With Screaming Fog I found the culprit on my site was having the 'Security Functions' activated in Sh404sef.
On = 403's - Off = no 403's but not all pages. Indexing did not seem to be affected either way but need to test that further.

I had a reply from the developers, they said that the site is crawled too fast, below is their response and it worked fine for me.

In the premium version you can slow down the crawl rate under 'speed' in the configuration. In the free lite version, you can crawl the site and then right click on any URLs with a 403 response and press 're-spider'. The server will generally then allow you to crawl these pages (and return a 200 ok response) as you're not requesting too many at once, so you might have to re-spider them individually.

Hope this helps

Thanks,
Handy to know for testing other sites that I may not have admin access for. I can crawl this one successfully if speed set to 1 for URI with security enabled in Sh404sef.
Cheers,
Dave

Thats great, glad it was helpful for you

The speed setting of 1 didn't work for me, but I've got administrator access and I whitelisted my IP address in the security settings of sh404 and that worked.

i have never this kind of issue.

but up to my knowledge & i have seen the refrence site...
might be useful

http://www.checkupdown.com/status/E403.html

thanks

Since it affects the Screaming Frog results, does anyone know if this also affects Google-bot? I presume that Screaming Frog and Google-bot uses a similar technique when crawling a specific domain.

I will try to fetch as Google-bot to see if there is 4xx in Google Webmaster Tools.

In your Google Webmaster Tools indicated HTTP 403 errors? It's in Webmaster Tools you should worry. In cases of websites made with CMS Wrdpress there is a pluging SEO that automatically solves this problem, but never seen in Joomla. Does anyone know if there pluging SEO to install in joomla? Mey name is Paulo Sebin, working with SEO in Brazil. I have attended interview about what is SEO (o que é SEO), on vimeo PT and have anything I would like to know about SEO for joomla if there is special pluging for this CMS.

Have you looked at the sh404SEF extension?

Desmotech wrote:With Screaming Fog I found the culprit on my site was having the 'Security Functions' activated in Sh404sef.
On = 403's - Off = no 403's but not all pages. Indexing did not seem to be affected either way but need to test that further.

actually am using Joomla 3 and Sh404sef, I had 59 403 Errors, after I switch Activate security functions Sh404sef control to No all 59 403 errors have gone for good
Just don't know how I will do without You, it has been a week now living without a sleep!