googlebots and high cpu usage.

Discuss Search Engine Optimization in relation to Joomla! 3.x. This forum will also have discussions on SEF/SEO Joomla! 3.x extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Windows Defender SmartScreen Issues <-- please read this if using Windows 10.
Locked
ziso
Joomla! Fledgling
Joomla! Fledgling
Posts: 2
Joined: Mon Jul 30, 2018 10:06 am

googlebots and high cpu usage.

Post by ziso » Mon Jul 30, 2018 10:21 am

Dear Joomla support i am running joomla 3.8.10 stable i noticed in my plesk panel that i have high cpu usage
in my logs i see that address Joomla! 3.8.10 66.249.69.5 which is a googlebot ask this webpage
GET /index.php?searchword=%CE%B1%CF%80%CE%AC%CE%BD%CF%84%CE%B7%CF%83%CE%B7&searchphrase=any&option=com_search&limitstart=25380 HTTP/1.0

the %CE%B1%CF%80%CE% is because my site is in greek
anyway i have several 100s of this that are using searchword or searchprase term and use the search results for crawling my website? the problem is that this is creating request that are to slow in that capacity the bot is asking on..is there a way to prevent it from crawling my searchpage?i still want my site to be crawled but not by this way. Is thera a solution via htacces robots or webmaster tools? Any idea would help.
Regards Ziso

Marcus Stafford
Joomla! Explorer
Joomla! Explorer
Posts: 308
Joined: Tue Apr 17, 2007 8:50 am
Location: Norwich, England
Contact:

Re: googlebots and high cpu usage.

Post by Marcus Stafford » Wed Aug 01, 2018 8:04 am

You can slow the crawl rate from within Google Console (Webmaster Tools).
Joomla! Managed Hosting, Web Design, Hack Repairs and Consulting ~ https://www.wintercorn.com

User avatar
digitalkube
Joomla! Apprentice
Joomla! Apprentice
Posts: 24
Joined: Sat Aug 11, 2018 6:32 pm
Location: India
Contact:

Re: googlebots and high cpu usage.

Post by digitalkube » Sat Aug 11, 2018 6:44 pm

Your best bet is to limit the crawl rate. To do so, follow the steps:
  1. On the Search Console Home page, click the site that you want.
  2. Click the gear icon and then click Site Settings.
  3. In the Crawl rate section, select the option you want and then limit the crawl rate as desired.
Keep in mind that the new crawl rate will be valid for 90 days so you'll have to adjust it again after this period.

ziso
Joomla! Fledgling
Joomla! Fledgling
Posts: 2
Joined: Mon Jul 30, 2018 10:06 am

Re: googlebots and high cpu usage.

Post by ziso » Sat Aug 18, 2018 4:42 am

so there is no way to hide exclude the index.php?option=com_search&view=search from robots or a way to add noindex no follow in the search results of joomla? i mean is there a way to add an index no follow in the search results of my page? or prevent system search to search particular words? i dont want this content to be searched at all

User avatar
digitalkube
Joomla! Apprentice
Joomla! Apprentice
Posts: 24
Joined: Sat Aug 11, 2018 6:32 pm
Location: India
Contact:

Re: googlebots and high cpu usage.

Post by digitalkube » Sun Aug 19, 2018 3:31 am

ziso wrote:
Sat Aug 18, 2018 4:42 am
so there is no way to hide exclude the index.php?option=com_search&view=search from robots or a way to add noindex no follow in the search results of joomla? i mean is there a way to add an index no follow in the search results of my page? or prevent system search to search particular words? i dont want this content to be searched at all
The only way to add noindex, nofollow to the particular file is by going to the file itself and add the meta to its head. This works in the case of static files only that come with the Joomla core. I didn't suggest it because you should never make changes to the core files directly.


Locked

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 3.x”