Unneeded bots a resource hog for Joomla?

[Slackervaara]

Since I started with my site I have had resource problem and slow site. With my two last web hosts with shared hosting my users could be greeted with the resource limit page and not the intended page rather often. AARGH!
Thanks to Cpanel and Awstats and Robot/Spiders visitors I found out that the culprit were: AhrefsBot (10 x more GB than GoogleBot), sem rush (like first). MJ12bot och BLEXBot.
When I added statements like this the problems were gone:
SetEnvIf user-agent "AhrefsBot" stayout=1
Ideally webhosts with shared hosting should ban uneeded bots to avoid resource limits.
The image below shows what happened after certain bots were banned - the right side of picture below

webbresurs.png

[Webdongle]

Have you considered changing Host?

[Slackervaara]

No, because there are not so many more pricy choices, but with this the problems are gone.

[Webdongle]

If their that lax what other things are they missing? You get what you pay for.

[sozzled]

Not the responsibility of webhosting companies. Do what most people do when they don't want search engine 'bots consuming site owners' bandwidth: block the 'bots with .htaccess rules. Hundreds of articles on the 'net that will show you what to do.

[Slackervaara]

I am a bit angry my first webhost told me you have too many posts and your site is spammed. This was not true as the main culprite was bots that stole the resources. My first host even shut down my site so I even was not able to back up and move. They have experts why does they not inform about this. Bots have slowing down my site for 18 years now.

[sozzled]

OK, so if 'bots are needlessly consuming host bandwidth—or, rather, how much bandwidth is available under your hosting plan—block the 'bots. I block these things with .htaccess and I don't have this problem.

[Slackervaara]

Bandwith is no big issue, but seem to diminish from 40 GB/month to 10 GB. The big issue is the slow loading of pages that can give PHP-errors and blank page. Worst is if there is resource limits that give an error page like this and can happen in worst case 2000 times a day.:

Resource-Limit.png

People get very irritated and wants a better and more expensive web hotel.

[Slackervaara]

I have seen big changes after banning bots:
RAM now 75 MB earlier peaks of 800 MB
CPU is low and never tops at 100 %
No faults now earlier up to 2000 a day.
Bandwith: 10 GB instead of 40 GB month
Earlier 250-750 MB cache in the morning now 25 MB
Redirect errors a day: Earlier 15 pages but now 5.
Visitors earlier 2000-6000 now approx 75-200
Page speed GTmetrix mobile/PC= now 99 %/100 % earlier 30 %/60 %.
I think cacheing the bots and enabling redirectingerrors makes things worse and consume resources also.
My site is pretty big 10 000 articles and 500 000 forumposts.