Advertisement

Trying to calculate how much crawling needed

Discussion regarding Joomla! 4.x Performance issues.

Moderator: General Support Moderators

Forum rules
Forum Rules
Absolute Beginner's Guide to Joomla! <-- please read before posting, this means YOU.
Forum Post Assistant - If you are serious about wanting help, you will use this tool to help you post.
Windows Defender SmartScreen Issues <-- please read this if using Windows 10.
Post Reply
User avatar
Slackervaara
Joomla! Ace
Joomla! Ace
Posts: 1124
Joined: Sat Aug 13, 2011 6:27 am

Trying to calculate how much crawling needed

Post by Slackervaara » Mon Jul 01, 2024 9:31 am

I am planning this month to calculate how much crawling for my site is actually needed. I am doing that by noting MB in database for com_content and com_jcomments that increases for my site. Then via Cpanel I can see how big the crawling is in GB. The problem is how much bigger is the article when browsed compared with in the database? Also things like CSS and javascript adds to article size.

Advertisement
User avatar
AMurray
Joomla! Exemplar
Joomla! Exemplar
Posts: 9985
Joined: Sat Feb 13, 2010 7:35 am
Location: Australia

Re: Trying to calculate how much crawling needed

Post by AMurray » Mon Jul 01, 2024 9:59 am

What do you mean by the term "crawling"? Do you mean data usage?

Search engines "crawl" a site when indexing them. I've never heard the term used out side of what search engines do.

There is one way to tell a web page's size; your browser (e.g. Google Chrome) tells you if you mousever the open browser tabs (or pinned tabs), it tells you how much each page is consuming in memory. Not exactly data usage (as in internet data quota) but it's a start....?
Regards - A Murray
General Support Moderator

User avatar
Slackervaara
Joomla! Ace
Joomla! Ace
Posts: 1124
Joined: Sat Aug 13, 2011 6:27 am

Re: Trying to calculate how much crawling needed

Post by Slackervaara » Mon Jul 01, 2024 10:32 am

With crawling I mean bandwith for robots for my site in GB per month. Then my increase in articles and comments will indicate the actual size increase of bandwidth. After I calculated this I will write to EU and demand smarter crawling of Google and Bing and only for new or changed material and not constantly the entire site. Webb hosts should automatically ban bad bots. This to decrease bandwith and the need for electricity for servers and decrease costs for smaller sites that dont are aware of this problem. Thanks for your suggestion about memory check.

Advertisement

Post Reply

Return to “Performance - Joomla! 4.x”