Expired URLs are attacking our website

[vivatech]

Hello Everyone:

I am seeking community’s help.

Something quite strange is happening to our company’s website at the moment. The site is recently migrated from version 2.5 to 3.4.

First of all, I noticed that our Google rating has dropped dramatically (we were rated well before migration). During my investigation this week; I noticed that several hundreds of expired URLs have been accumulated under “Redirect Manager” at the back-end. Some of these expired URLs are outdated URLs of our website, others are not related to our site at all, or appeared as sub-directories of our website, by judging the names, some are pornography site URLs.

I am in communication with our site hosting company, they are willing to help and doing the best they can, but I have feeling this is outside of their expertise, and this is why I am seeking experts in Joomla community to help.

I tried manually deleting expired URLs, but new URLs are re-appearing in the speed of a couple hundreds per day at the moment. It seems the more I delete, the more are showing up.

It is in my opinion that our site has been attacked. I will be very much appreciated to hear about Joomla community experts' advices on how to deal with this problem?

Regards,

VivaTech

[itoctopus]

Have you checked your Apache logs to see how these URLs are being added?

Also, a good idea would be to disable the "Redirect Manager" plugin (this extension will be deprecated in future versions of Joomla anyway).

Another thing to consider is whether the drop in Google rankings is related to a core modification that no longer exists. Usually, when we do a migration of a large website, we keep a watchful eye on the 404 errors in Google Webmaster tools. If you're seeing a large number of 404s, then you should investigate why this is happening.

[vivatech]

Hello Itoctopus:

Thanks for your quick and valuable comments.

Please forgive my ignorance, where can I find "Apache logs"? Does my web-hosting company have it? I have FTP set-up already connected to the web hosting company's server, is it where I can find these log files? Please advice.

Regarding to your comments about 404 errors in Google Webmaster tools, yes, I did see large numbers of 404 error pages there. Most of them are URLs of pages that are work-in-progress, and they are not ready to be published on the web. I am not sure how Google got hold of them. Please advice me on this also.

Thanks again and I look forwards to hearing from you.

Regards,

VivaTech

[itoctopus]

The Apache logs can be accessed from your cPanel account under logs.

A large number of 404s can lead to a reduced traffic from Google. You should ensure that these work-in-progress pages are not indexed by Google by adding a rule in your robots.txt file or by ensuring, at the individual article settings level, that the article is not indexed.

[vivatech]

Hello Itoctopus:

Thanks again. I will look for the log files, and will report back.

Regards,

VivaTech

[kappadev]

Hi vivatech, i seem to have exactly the same problem.
Did you find what is causing it?

thx

[Bernard T]

First thing to do, as suggested before, is to search the access logs to find who is opening those URLs. When that fact is confirmed you will have much better chance to fix it.

Let us know what you have found.

[vivatech]

Dear all:

It is a relief to know that I am not the only one having this problem.

Mine are not solved, but alleviated. Root causes are still unclear. Some advices from my hosting company are as follows:

“These links must have been at some point active on your site and somebody must be clicking them or referencing them in another way - that's why they appear. The only way to stop this would be to either find out who is using them and tell them to stop (which I'm afraid we have no way of tracking) or create redirects for those links - so that at least they don't throw a 404 page. You can also just ignore them (they don't pose any danger to the site), but that might not be such a good idea if their amount increases - the redirect table in the database is read by the server every time no matter what, just to check, so if they pile up, it may cause the site to slow down “.

But I have no way to find out who the “somebody” is.

I tried followings thus far:
1. Analyzed the type of expired URLs, and it comes into three categories:
a). Current work-in-progress URLs that I have been working on new designs
b). Old URLs that were published, but are no longer in use
c). Pornographic URLs
2. Base on the result of my analysis in Step 1, I added rules in the robots.txt file to disallow certain files to be crawled by search engines
3. Signed up with Google Analytics account and Yahoo/Bing Site Tool account, manually submitted URL removal request for each 404 error page occurred one by one.
4. Step 2 & 3 helped to alleviate attacking by expired URLs under category 1-a), and 1-b); however, the expired URLs under category 1-c), the pornographic URLs are continuously attacking / flooding “Redirect Manager” folder of the back-end. I have not find a way to deal with it except logging into the back-end each day to manually remove them.

It is only in my opinion (I am not a Joomla expert, only a user), there are security issues or weak points within Joomla version +3, my site under version 2.5 performs smoothly well and has no problem at all.

I hope my comments are useful; I would appreciate a continuously investigation or discussion about this problem if of any interests.

Regards,

VivaTech

[Bernard T]

Hi Vivatech,

you still didn't follow previous poster's and my instructions!

1. Go to "access logs" and find out which IP's are trying to open this non-existing links. Post some IP examples here.

2. Post FPA report

It is quite possible that your website is SEO spam links poisoned, which usually means some malicious code (template, extension or modified .htaccess).

Follow first two suggestions and we can try to help you further.

[mandville]

I think that this may be a two scenario at play.

Your old site urls are appearing as Google etc are looking for them and they can't be found hence the redirect plugin is offering the chance for you to redirect them.
The porn sites could be classic referer spam. Sites that visit your in the hope of appearing in public accessible site logs / stats and therefore being backlink to you.
My suggestion is disable the plugin and purge the field feom the Sql.
If you have cpanel access then you can set up or disable Stats viewing .