MS Bing web crawler out of control causing our site to go down

Discussion regarding Joomla! 3.x Performance issues.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Windows Defender SmartScreen Issues <-- please read this if using Windows 10.
Locked
konlungkao
Joomla! Guru
Joomla! Guru
Posts: 732
Joined: Mon Sep 12, 2005 3:24 pm
Location: Thai
Contact:
Contact konlungkao

MS Bing web crawler out of control causing our site to go down

Post by konlungkao » Fri Jan 11, 2019 1:03 am

MS Bing web crawler out of control causing our site to go down

how to fix it
You do not have the required permissions to view the files attached to this post.
รับทำเว็บ joomla http://www.colorpack.co.th | ถามตอบ ปัญหา Joomla ฟรี http://joomlachannel.com
Top
waarnemer
Joomla! Hero
Joomla! Hero
Posts: 2954
Joined: Sun May 04, 2008 12:37 pm

Re: MS Bing web crawler out of control causing our site to go down

Post by waarnemer » Fri Jan 11, 2019 8:52 am

I don't know if your site is brought down, but BING is crawling pages (or rather, tries to crawl) that no longer exist (hence all 404's).

Did you use to have an archive published? looking at the urls that it is trying to crawl.....

Second.. you use cloudflare but did not install mod_cloudflare on your server... therefore all visiting IP adresses are cloudflare IP addresses..
https://support.cloudflare.com/hc/en-us ... loudflare-

You do have bingbot in your robots.txt.. but using cloudflare, are you sure cloudflare has the correct pages (and robots.txt) in cache?... you may want to flush all to be sure it gets all current and active pages.
Top
Locked

Return to “Performance - Joomla! 3.x”