Jumping through hoops in replying to 'bot-generated emails from the JED

[sozzled]

Greetings,

I don't want to make a big issue of a small problem that I encountered today, except to explain the difficulty I had with replying to an email I received (see attached image):

jedBrokenLink_email.jpg

Aside from the matters raised by that email (which are unimportant as far as this discussion is concerned), I replied to the email—no realising that the sender's address was [email protected]. I assume that my emailed reply will simply disappear into a trash bin.

Therefore, some time after realising my mistake, I had to login to extensions.joomla.org and "raise a ticket". The total round-trip of my time was about 15 minutes (excluding the time I spent checking whether the original report was actually factual) to reply to that email generated by some 'bot.

It may be a better idea if, instead of having to "raise a ticket", quote the contents of the email I received (together with my reply), we could simply reply to these system-generated emails via email?

Just throwing this out there for discussion, that's all.

[anibal_sanchez]

Hi,

These are early notices about broken links. The developer receives at least three of them in a period of one month before unpublishing the extension.

If the developer doesn't fix the link in time, then the extension is finally unpublished and then a ticket must be submitted.

The need for replying or submitting a ticket is reserved for the cases where the developer doesn't receive the emails.

Best Regards

[sozzled]

I don't really understand. From the sounds of things, if a developer receives one or two emails then they can ignore them; these automatically generated emails are "early warnings" (of possibly nothing to worry about). If a developer receives three emails then there's a problem.

I'm sorry but it doesn't make sense. A 'bot generates an email (or two or three) but there's no human who's checking if these automatic emails are legitimate. On the other hand, a developer cannot reply to these automatically generated emails and needs to login to the JED portal and "raise a ticket". Is that correct?

In my case I replied to the email (by creating a ticket) but I have not heard back apart from an automated "We have received your ticket" reply.

[anibal_sanchez]

Hi,

The LinkChecker detects broken links, so the developer must fix them as soon as possible.

The workflow only sends three notices just in case the developer misses the first notices. In the past, the checker unpublished the extension after the first fail.

Best Regards

[sozzled]

Following up, I received a reply from Denes this morning. A public thank you to him for his reply.

To put things into context, he wrote

Michael, sorry for the confusion. You got the email from the automated script we are using AGAIN for several weeks by now (read this: https://volunteers.joomla.org/teams/ext ... -27th-2020). The script has been written years ago, used for a while, then "parked", and we know that is NOT perfect - we are aware of a number of problems, and our devs are continuously improving it. If you think, you found a bug, or you think you have an idea how we possibly could improve it, please open an issue here: https://github.com/joomla/jed-issues/issues. Please, check before that if your URL's posted in JED are perfectly matching your real URL's, from our experience one of causes of the false positive alerts is a redirect on your site. For example if you posted originally a http:// link, and now the site is switched to https://, with a redirect in place to map the http:// links to https:// ones, our script is susceptible to trigger false positives. Anyway, the extension will be unpublished ONLY after being found 3 times in row in error, so you have all the time in the world to check if the links are OK. We needed to re-implement this because of massive number of broken links we found in JED - only in this weekend 200+ problems where reported. This is a mission impossible task for the fistful of people volunteering here to hunt down these by hand. So, we apologize for eventual inconveniences, and looking forward for your valuable feedback to make this script to be as accurate as possible.

I understand the JED team's near-impossible task of fixing "broken links" but I wonder whether it should be the JED team's mission to check these things at all?

As far as possible, JED listings are taken "on faith". That is, a few standard checks are made to verify that a listing complies with the JED terms-of-use (e.g. licensing, naming, copyright notices, etc.) built into the JED checker. Other than those things, it's largely a matter of caveat emptor.

If any other member of the community discovers some problem with a JED listing they have the ability to report the listing to the JED team who may then initiate enquiries to see whether the report is legitimate or not. It's not really a case of the JED team arbitrating on commercial disputes; it's a case of ensuring that listings satisfy the basic requirements.

Anyway, while the system is not perfect and the methodology could be improved, I accept the explanation.

[sozzled]

I have received another bogus "broken links" notice from the automated link checker (similar to the case I reported in the opening post). I have changed nothing; the links in the JED listing for my extensions are the same as they have always been. I think it's a waste of time creating a "ticket" and having a JED team member reply. I don't think it's a productive use of our time or the JED team's time to have to report each case when these bogus reports are generated by the broken link checking system. Please fix this annoyance: preferably disable the automated link checker.

I don't think the JED team should be wasting their time or ours on this.

[anibal_sanchez]

Hi,

If you are receiving the Broken Link notice, you must check the links of your extension.

If they are Ok, please, send us a ticket to check the logs of your extension and confirm the issue.

PD: Of course, if the report is not solved timely, the extension will be unpublished.

Best Regards.

[sozzled]

*grumble*. What a complete waste of time. The email is the same email I received three weeks ago. I raised a ticket (and received a lovely reply) and there was nothing wrong then. We closed the ticket. I can give you the ticket number if you like.

Are you saying that every time, every couple of weeks, when we get a robodebt letter, we have to raise a job ticket? Are you saying that the burden of proof for your broken system lies with us? That's absurd!

[anibal_sanchez]

Hi,

If you send us a ticket we can support your case until we find a solution, we can help to solve your issue or improve the link checker. Other users are already sending suggestions and improving the process to check the links that are published ensuring that we publish quality listings.

Of course, you are free to proceed in the best way that suits you.

Best Regards

[sozzled]

Let's start with how to reply to the system-generated email first. We can't use email to reply to the email!

See the following image of the latest email I received:

jedBrokenLink_email200710.jpg

We cannot use the "ticketing system" to attach the email (or the text of the email) to help the JED team members understand the nature of the issue. We have to spend time finding where the support ticket link is located and then write something like this:

I received an email today that reads:

"Hello <my-name>,

"The Joomla Extensions Directory automated link check noticed that the <listing-name> has links that are broken.

"Please ensure that the server is available and that the links (Project homepage, Download, Demo, Documentation, Support and License) are correct. We will re-check this link in a few days; if it is still unavailable, your extension will be unpublished.

"Update the extension links ASAP.

"If the extension is already unpublished, please, log in and open a support ticket with this link: Listing / UPDATE - Unpublished extension, broken links fixed.

"Best Regards,
The JED Team"

Because the email was sent from a "noreply" address, I am replying here.

I checked the JED listing for this extension and all the links on the listing are correct and active as far as I can tell.

If there is a “broken link” somewhere, I should like to know which link(s) are broken. It’s possible that the JED automated link checker reported a false positive “broken link” when it ran.

I also tested the extension with JED Checker v2.1.3 which reported no problems.

Thank you

<my name>

That's what I wrote in ticket number/code REPORTS-XJX5MO3R9E. This took me 25 minutes: to locate the listing on the JED, look at all the URL links, test each one, run the JED Checker, login to the JED, locate where to create a ticket, find the appropriate subject line, write the reply and click send.

Of course, you are free to proceed in the best way that suits you.

Let me think about this. It doesn't matter a lot to my business bottom-line if the listing in question is published or not published; I get very few download requests for that particular extension. It does matter to my total business bottom-line if any of my extensions are unpublished by the JED because I cannot add any new products to the JED until questions about unpublished extensions are resolved. That's the problem!

It's not a problem about whether one listing is unpublished. It's a problem when, because one listing is unpublished, this affects all extensions the developer has listed on the JED. Can you understand that? Let's say a developer has 10 extensions listed on the JED (and, for whatever reason, one of those extensions is de-listed by the JED team/process). This means that the developer is unable to add any new extensions to the JED under their account until they have either removed or fixed the one extension that has a problem with it!

I'm not writing here to complain about the injustice that I feel. I'm writing here to express my concerns on behalf of other extension developers and to demonstrate my sympathy for the imposition of additional hours that the JED team members spend in responding to these false alarms.

If you send us a ticket we can support your case until we find a solution, we can help to solve your issue or improve the link checker. Other users are already sending suggestions ...

I have previously sent you my suggestions to improve the process (see ticket number/code REPORTS-XJX5MO3R9E).

I do not understand what business it is of the JED to [automatically] verify the correctness of information, including links, in the JED. I would remind everyone to look at the JED Terms of Service (Section 11)

The listings and reviews listed in the JED have been submitted by the community and are for general informational purposes only. Listings do not constitute or imply endorsement, recommendation, or favoring by Joomla or OSM.

Because the listings and reviews are provided by others, Joomla, OSM, and the JED team cannot be held liable for accuracy of the information. Visitors wishing to verify that the information is correct should contact the parties responsible for creating the content of the resource.

I agree: the JED should not be held accountable for the accuracy of the information. If a listing fails to meet the requirements of JED ToS sections 4.1(c) and 4.3(a) and, further, if listings infringe on JED's terms of service (i.e. " you are free to proceed in the best way that suits you") and the provisions of JED ToS section 7 are enforced, the burden of proof falls on the developer/advertiser!

I'm saying that the burden of proof, that the automated link-checking system is reliable, falls on the JED team and not on the developers to disprove those things detected by that system. As far as the JED is concerned, we have an automated system that has taken over from human beings—because we humans have not objected to allowing this to happen; we could probably describe the issue as JEDinator.

The best solution is to kill off JEDinator before it completely destroys our trust in the system.

This forum category "extensions.joomla.org - Feedback/Information" receives an unusually high number of complaints. These complaints will not stop because developers will use the job ticket system to express their displeasure; these complaints will increase because of the unnecessary hurdles that the JED system imposes on people who try to use it.

I will spend the extra 25 minutes of my time checking my listing (even though I have not changed anything in the past three weeks), login to the JED, find where the support ticket link is located—have you tried?—and use the postage stamp-sized interface to write my reply. I will then wait for a day or two before one of the overworked JED team members receives my reply and writes back to me saying, in effect, "all is well."

Everyone's time is wasted!

[brian]

Is it just for one of your extensions?
If so then please share the name and people can try to work out why it happens (as you dont seem willing to trust the JED team to do that)

[sozzled]

Thanks, Brian, but I could get into strife for posting the name(s) of any of my extensions: y'know, the "self-promotional" rule of the forum. But, to answer your question as simply as I can: yes. It's just one of my extensions (at the moment).

This is the problem, as far as I can work out: I have an extension, listed on the JED, that has "links". The automated checking tool believes that one (or more) of those "links" is broken. If you went to the JED listing, you would see something like this:

jedLinks.png

There are eight links there. I have numbered these links 1 through 8.

The text of the email, from the JED automated checker states, "Please ensure that the server is available and that the links (Project homepage, Download, Demo, Documentation, Support and License) are correct. We will re-check this link in a few days; if it is still unavailable, your extension will be unpublished."

That's what the email states, word for word. The email does not specify which "link" is "broken"; it just says that I have to check all of them and, regardless of what I find, if the automated checker finds something is wrong, the system will unpublish my extension. Ergo, JEDinator!

After unpublishing my extension (through no fault of mine, BTW), I am then required to submit a ticket (which is, itself, is a non-trivial exercise), with the subject "Listing / UPDATE - Unpublished extension, broken links fixed." ... even though nothing was broken to begin with.

In other words, the burden of proof is on us to disprove the findings of the automated link checker!

I don't mind doing that occasionally. Once every few months, perhaps. But this is now the second time this has occurred in the space of three weeks and, unless we start objecting to this waste of our time, JEDinator will turn into an unstoppable monster. It's not our job to fix this broken system. I say it's not within the JED's charter to be automatically checking for these things in the first place.

It's also worth looking at what @mbabker wrote on this matter:

For some reason, people seem to put way too much weight on having a listing on the Joomla Extensions Directory. Like it's a badge of honor or something. And for some reason, historically the JED has had over-reaching policies on what links are allowed, what text is displayed on those pages, and what text is displayed on other pages listed on the website. There was once upon a time that you couldn't link from the JED to a website that included code released under a license that wasn't the GPL, luckily some asininely stupid rules seem to have been relaxed over the years.

If you want JED to be meaningful, then about 85% of the automated garbage needs to go away, and a higher bar to entry needs to be enforced. Seriously, you want the site to be filled with quality products, but to get listed you have to provide a link that has a 200 response (I think I saw at one point the auto link checker chokes on 30x responses, and heaven forbid your site has a 500 response when the cron runs), a bunch of files with a GPL license and a "defined or die"... err, "defined or exit" (because 2020 PC terms)... check, and that's about it. There has never, in at least the 10 years I've been stuck in these parts, been any form of human related quality assurance to help developers and instead way too much time is spent on trying to write more automated tools to make sure variables are defined in the right spot in the right file.

It's not just me who's complaining but I may be the only one who's prepared to stand up and publicly object to these things.

I don't think anything will change because of my protests or because—thanks, Brian, for your offer to help—others may find the cause of why one of my extensions has been targeted by JEDinator. Something may change if one of two things occurs: (a) more people tell their stories here on the forum, and (b) the JED team members, exhausted by spending hours of their time dealing with the flood of tickets, walk out and leave no-one to answer our requests.

[brian]

I offered to help - you turned it down - nothing more anyone can do for you

[sozzled]

I didn't turn down your offer, Brian. I wrote that I would be difficult and awkward for me to display the name of the extension, in public (on this forum) that has been hit by the JEDinator. If you are interested in the details of the matter I can send the information to you via PM.

My question is a simple one—well, I have two questions:

What purpose is being served by having a broken system target honest extension developers requiring us—honest people who have our products listed on the JED—to check our listings every few weeks to "ensure" that these links are working when JEDinator runs and trawls things?

How many genuine cases of broken links are discovered? In the past few days I saw the number of extensions go from 7,494 to 7,504 and today there are 7,502. Two years ago (according to this information) there were "more than 7,700 extensions" listed on the JED; today there are 7,500 or thereabouts. Is it really the responsibility of the JED team to check the information given the disclaimer made in Section 11 of the terms of service that OSM, the JED team and the Joomla! project are not responsible for the accuracy of the JED content?

I'm mystified why JEDinator has been accepted by the community of third-party extension developers whose products are listed on the JED unless there's a real problem here and the end users of the service (i.e. people who try to obtain products/information from the JED) are complaining en masse about how bad the JED is. If that were the situation then a case could be made for the JED team to actively ping extensions listed on the JED but I can't find evidence of any wrongdoing by people who list their extensions on it.

[brian]

If you're not prepared to accept genuine offers of help I can only assume that you are trolling

[sozzled]

Huh? Where did that come from? If you want to help then I can give you the name of the extension via PM but I can't post the information in public, on the forum, because it would be considered to be self-promoting.

Question to the forum moderators: Is it considered self-promoting to post the name of one of my extensions, in public on the forum, relating to this discussion topic?

[toivo]

Please post the details by all means, as @brian suggested.

[sozzled]

Thanks, @toivo. I did not want to assume that I had the automatic right be able to do that. I wanted to be sure that I would not get into trouble for following a suggestion from someone who is not a forum moderator.

The extension in question is this: https://extensions.joomla.org/extension ... or-kunena/

I should be grateful for any help I can get, including an explanation about which of the links in that listing are broken.

[anibal_sanchez]

Hi,

Again, the only way to contact the JED support is with a ticket. We can't disclose with you the extension information, internal data or logs here in the forum.

Of course, if you prefer to discuss your case with other users in the forum, you are totally free to do it. From my side, I don't have anything else to say.

Best regards,

[brian]

I would definitely
1. kill the statcounter script as that definitely has the potential to delay the load of your site that a link checker might think the page is down.
2. Fix the missing image on the demo
3. Check the very poor perfomance on the support page coming from the gantry js

[sozzled]

Thank you, Brian, for taking the time.

Hmmmm ...

... so, basically, you're saying that my website skills are not up much and that it's my fault that JEDinator doesn't get a 200 response code? These things in spite of the fact that you had no real drama in clicking all the links yourself. Hmmm ... not sure what to make from that.

The websites were designed by a human being for other human beings to use. It was not designed by a robot for other robots. As a human being, who designed the websites, I should be forgiven for making a few mistakes, wouldn't you agree? Should I be punished because JEDinator isn't as tolerant of our humanness?

1) The Statcounter analytic measuring tool is used by 2 million websites around the world;
2) What "broken image" on the demo site?
3) Very poor performance on the support page? Like how? The support page—I assume you mean my forum—loads under 2 seconds.

But, as you probably know, I'm not the only person with these JEDinator issues. I wasn't trolling the forum; I wasn't making things up. I asked for help in good faith. Thank you for your help (and thanks for spending a little over 10 minutes looking around my websites). As you discovered, all of the JED listing links that you followed worked.

[brian]

Failed to load resource: the server responded with a status of 404 (Not Found) rss.png

======================================================
index.php:720 [Violation] Avoid using document.write(). https://developers.google.com/web/updat ... ment-write
(anonymous) @ index.php:720
index.php:720 [Violation] Parser was blocked due to document.write(<script>)
(anonymous) @ index.php:720
index.php:720 A parser-blocking, cross site (i.e. different eTLD+1) script, https://secure.statcounter.com/counter/counter.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.
(anonymous) @ index.php:720
index.php:720 [Violation] Parser was blocked due to document.write(<script>)
(anonymous) @ index.php:720
index.php:720 A parser-blocking, cross site (i.e. different eTLD+1) script, https://secure.statcounter.com/counter/counter.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message. See https://www.chromestatus.com/feature/5718547946799104 for more details.
(anonymous) @ index.php:720
jquery.min.js?5049de2e6d223a646845f9aed2b333ab:2 [Violation] 'DOMContentLoaded' handler took 247ms
[Violation] Forced reflow while executing JavaScript took 71ms

=============================================
main.js:1 [Violation] Added synchronous DOM mutation listener to a 'DOMSubtreeModified' event. Consider using MutationObserver to make the page more responsive.
s @ main.js:1
(anonymous) @ main.js:1
e.exports @ main.js:1
forEach @ main.js:1
on @ main.js:1
attachMutationEvent @ main.js:1
attach @ main.js:1
constructor @ main.js:1
(anonymous) @ main.js:1
t @ main.js:1
[Violation] Forced reflow while executing JavaScript took 65ms
all.js:5 [Violation] 'setTimeout' handler took 86ms
[Violation] Forced reflow while executing JavaScript took 82ms
main.js:1 [Violation] Added synchronous DOM mutation listener to a 'DOMSubtreeModified' event. Consider using MutationObserver to make the page more responsive.
s @ main.js:1
(anonymous) @ main.js:1
e.exports @ main.js:1
forEach @ main.js:1
on @ main.js:1
attachMutationEvent @ main.js:1
(anonymous) @ main.js:1
(anonymous) @ main.js:1
[Violation] Forced reflow while executing JavaScript took 97ms

[sozzled]

Thanks, Brian.

I disabled RSS from the demo site (it isn't needed there).

I've decided to take the line of least resistance in this matter. Tomorrow I will just unpublish two of my extensions from the JED. As more of my extensions are JEDinated, I will probably unpublish them, too.

Before I commercialised all of my extensions that I had listed on the JED, the module we're discussing had been downloaded over 600 times. In the past three years since I commercialised everything, the module has been downloaded about ten times. It's not worth anything to me to advertise it on the JED. If people want it they can purchase it; if they don't that's OK with me, too.

[sozzled]

I've made a few changes to my demo site (not that it matters a lot in the grand scheme of things) which improve the site's performance; in spite of the JED team's reluctance to admit that JEDinator is a waste of everyone's time and the tone of the emails we receive from it is threatening, I've unpublished two of my extensions from the JED. At least, by unpublishing those listings, I'm saving a bit of bandwidth and JEDinator will probably run quicker because there are fewer things for it to check

I mentioned yesterday,

In the past few days I saw the number of extensions go from 7,494 to 7,504 and today there are 7,502.

When I look at the JED now, there are 7,482—a reduction of twenty extensions since yesterday. So the effect of threatening people with de-listing their extensions on the JED is evidently being taken seriously and other third-party developers are likewise removing their content from it.

This matter is far from being resolved satisfactorily in my opinion. We evidently have a different view to the JED team who want to keep JEDinator running while my view is the opposite.

[brian]

> and other third-party developers are likewise removing their content from it.

More likely the link checker is working correctly and extensions that no longer exist are correctly removed

[jonBuckner1]

I feel sorry for you sozzled.

A link checker that says you have a broken link but will not tell what that link is... a genius way to annoy people.

It does seem that your efforts to make jed better fall on deaf and defiant ears

[sozzled]

Thanks, Jon.

What's worse is that, when discuss issues here and we receive responses like

... the only way to contact the JED support is with a ticket ... [but] if you prefer to discuss your case with other users in the forum, you are totally free to do it. From my side, I don't have anything else to say.

... this makes a mockery of using the forum, doesn't it?

I thought the purpose of this forum was to contact the JED team to provide feedback? (see screenshot below)

JEDfeedbackl.png

Apparently that's not the purpose. We have to jump through hoops ...

If the purpose of the forum is for us to have discussions among ourselves and not with the JED team then I think the forum should say that. The forum category should make it clear that this forum is not the place to provide feedback to the JED team and that the only way to contact the JED team (if you have ideas, questions or problems) is to use the ticket system.

[anibal_sanchez]

Just in case, it is missing of the conversion. Yes, you can contact the editors of the Extensions site here, as well as access infomation relating to this site. However, the team is not free to talk publicly about extensions specifics here (or in any other medium). The only way to manage your extension is via the site and sending tickets.

Best regards

[anibal_sanchez]

For the record, the same answer here:

Hi,

The LinkChecker tests the links three times in a month and sends emails to notify about the issue. If there is no change and the links can't be verified, then the extension is unpublished.

If you have checked the links and the links are working fine, then probably your server is blocking the JED Checker. The Link Checker runs from the server https://extensions.joomla.org/ (72.29.124.148 - joomla-extensions.directrouter.com) and it can be identified by the User Agent "Joomla-JED-LinkChecker".

Please, try to whitelist the agent or the IP on your firewall solution. In any case, send us a ticket to investigate the case and give you a hand checking the log files.

Best Regards