Page 1 of 1

MS Bing web crawler out of control causing our site to go down

Posted: Fri Jan 11, 2019 1:03 am
by konlungkao
MS Bing web crawler out of control causing our site to go down

how to fix it

Re: MS Bing web crawler out of control causing our site to go down

Posted: Fri Jan 11, 2019 8:52 am
by waarnemer
I don't know if your site is brought down, but BING is crawling pages (or rather, tries to crawl) that no longer exist (hence all 404's).

Did you use to have an archive published? looking at the urls that it is trying to crawl.....

Second.. you use cloudflare but did not install mod_cloudflare on your server... therefore all visiting IP adresses are cloudflare IP addresses..
https://support.cloudflare.com/hc/en-us ... loudflare-

You do have bingbot in your robots.txt.. but using cloudflare, are you sure cloudflare has the correct pages (and robots.txt) in cache?... you may want to flush all to be sure it gets all current and active pages.