Taming the Bing bot

Discuss Search Engine Optimization in relation to Joomla! 3.x. This forum will also have discussions on SEF/SEO Joomla! 3.x extensions.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Windows Defender SmartScreen Issues <-- please read this if using Windows 10.
Locked
User avatar
stevent
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 123
Joined: Thu Sep 15, 2005 6:05 pm
Location: Nijmegen, Netherlands
Contact:

Taming the Bing bot

Post by stevent » Sat Dec 02, 2017 10:28 am

The Bing bot is one of the worst bots there is. It's generating 1/3 of all traffic to my site (3000 hits over 24 hours), more than twice as much as the Google bot (1300 hits), though I've set the crawl rate for Bing enormously slow (In the webmaster tools as well in the robots.txt). It seems that the crawl delay is respected by each separate Bing server, but not as a whole. So Bing bot no. 1 could come every n seconds, but in the meantime Bing bot no. 2 comes along, then bot no. 3 etc.

What I've noticed is that Bing is not only using my sitemap (set up with OSMap), but it also tries to use any rss feed it encounters as a sitemap. When I look at the sitemaps in the Bing webmaster tools, url's with ?format=feed show up. If I remove these, they will appear again within days. And thus Bing seems to crawl more pages than needed.

I've setup Bing webmaster tools to ignore url parameters like 'format', but it doesn't seem te respect my preferences. I've also added a line to my robots.txt, but even that is ignored too:

Code: Select all

Disallow: /*?*format=
Has anyone an idea how to tame the Bing bot? I'm not paying my hosting provider to please Bing, especially since it only accounts for less than 1% of the referral traffic.

Locked

Return to “Search Engine Optimization (Joomla! SEO) in Joomla! 3.x”