Unpublished extension [again], immediately after email warning, bot IP change.

[eorisis]

I received an email warning that there are broken links and after further investigation I found out the following.

The Joomla-JED-LinkChecker bot changed from IP 72.29.124.155 to the IPv6 2602:fd32:1:6a6f:6f6d:6c61:7:1 and because of this it was no longer whitelisted on my servers. For that reason it got blocked.

This alone should not be a problem, but here the above bot visited my website about an hour ago as seen here (notice the time):

Apache log

Code: Select all

[...]
2602:fd32:1:6a6f:6f6d:6c61:7:1 - - [21/Jun/2023:16:43:56 +0300] "GET /applications/google-maps HTTP/1.1" 403 6730 "-" "Joomla-JED-LinkChecker"
2602:fd32:1:6a6f:6f6d:6c61:7:1 - - [21/Jun/2023:16:44:00 +0300] "GET /applications/google-maps/documentation HTTP/1.1" 403 6730 "-" "Joomla-JED-LinkChecker"
2602:fd32:1:6a6f:6f6d:6c61:7:1 - - [21/Jun/2023:16:44:04 +0300] "GET /contact/support HTTP/1.1" 403 6730 "-" "Joomla-JED-LinkChecker"
[...]

WAF log

Code: Select all

# Action Time: Wednesday 21, June 2023, 16:43:56::6451 (UTC +03:00 DST)
# Remote Hostname: 2602:fd32:1:6a6f:6f6d:6c61:7:1
# REMOTE_ADDR: 2602:fd32:1:6a6f:6f6d:6c61:7:1
# HTTP_USER_AGENT: Joomla-JED-LinkChecker
# REQUEST_URI: /applications/google-maps
# HTTP_CF_RAY: 7dacaef06b68e997-DFW
# HTTP_CF_IPCOUNTRY: US
# Country: United States
# Country Code: US
# Reason: Fake Joomla-JED-LinkChecker

There is no other visit before this for all of June, and none with this new IP before. Some minutes later I received the email warning about broken links, notice the time in the headers:

Code: Select all

Return-Path: <[email protected]>
Delivered-To: xxxxxxxxxxxxxx
Received: from xxxxxxxxxxxxxx
	by xxxxxxxxxxxxxx with LMTP
	id SOBOJCD/kmR61QMAK06oTw
	(envelope-from <[email protected]>)
	for <xxxxxxxxxxxxxx>; Wed, 21 Jun 2023 16:46:08 +0300
Received: from localhost (localhost.localdomain [127.0.0.1])
	by xxxxxxxxxxxxxx (Postfix) with ESMTP id 8B5325243041
	for <xxxxxxxxxxxxxx>; Wed, 21 Jun 2023 16:46:08 +0300 (EEST)
X-Virus-Scanned: Debian amavisd-new at xxxxxxxxxxxxxx
Authentication-Results: xxxxxxxxxxxxxx (amavisd-new);
	dkim=pass (1024-bit key) header.d=joomla.org header.b=HCKsFl/9;
	dkim=pass (1024-bit key) header.d=elasticemail.com header.b=IgOwy2Mx
Received: from xxxxxxxxxxxxxx ([127.0.0.1])
	by localhost (xxxxxxxxxxxxxx [127.0.0.1]) (amavisd-new, port 10024)
	with LMTP id rRvuO-i2_lqj for <xxxxxxxxxxxxxx>;
	Wed, 21 Jun 2023 16:46:08 +0300 (EEST)
[…]

No other email before that. But the extension is already unpublished. So what is happening is that the bot visits a url, when it gets any status other than 200, it immediately un-publishes the extension. This doesn't allow for the 30 days (or any time at all) to fix any possible issues. This is a problem combined with the IP change.

I whitelisted 2602:fd32:1:6a6f:6f6d:6c61:7:1 but I need a confirmation that this is the legit IP.
From my knowledge it comes from the same AS 17378 AS17378 TierPoint, LLC.

I submitted a ticket 5 minutes ago with code: LISTING-HURDE7501C
Please publish the extension back because we got a deeper problem here.
Thanks.

[mandville]

have you considered whitelist the BOT NAME instead of the revolving IP?
the IP revolves as a fraud deterrent

[eorisis]

What do you mean by "the IP revolves as a fraud deterrent" ?

The WAF uses a combination of User-Agent string and source IP. Whitelisting User-Agent string only provides no security.

[mandville]

ok then if you dont understand that situation, is it something that is happening in your WAF that is causing it as i am not aware of a flood of complaints over the change in IP causing 1000 of listings to be unpublished for 404 errors

[eorisis]

You haven't explained it well for me to understand, I can only guess. Do you mean that the JED bot keeps changing IP on purpose ? I can't think of a reason for that. this would only cause problems and have no benefit.

In my case there are no 404 errors. There are 403 as you saw above, as the firewall doesn't allow illegitimate connections from bots. Do you consider this a bug in the WAF if I understand you well ?

The other big problem is that the JED unpublished the extension immediately. I thought it sends emails as warnings and there is a period of about 30 days for the links to be fixed. Am I wrong ? If the JED unpublishes extensions immediately, then I should fear rebooting, updating or performing any other tasks on the server because it could at any moment lead to unpublished extensions. Do you consider this OK ?