Wrong overall rating score

Here you can contact the editors of our Extensions site, as well as access infomation relating to this site.

Moderator: JED Team

Forum rules
Forum Rules
READ ME <-- please read before posting, this means YOU.
Locked
qvsoft
Joomla! Apprentice
Joomla! Apprentice
Posts: 27
Joined: Wed Apr 15, 2009 7:22 am

Wrong overall rating score

Post by qvsoft » Mon Jul 20, 2015 7:34 am

Overall rating of extension on JED is wrong in some cases, it seems be lower for free extensions.
I see a report in tracker system sine Apr 2015 but not resolved until now, and also try to submit ticket to ask about this, but JED team referring me to this article: http://extensions.joomla.org/support/kn ... g-overview and it does not help.
I can show some examples here, when the extensions have the same rating score on each review, but overall rating is very different:

- Some time the overall rating start from 72%, some time from 92%. let's see these 2 extensions have the same 100% rating on each review, but overall rating is different:
http://extensions.joomla.org/extensions ... no-doubles -> 92 + 92 + 92 + 92 = 72%
http://extensions.joomla.org/extensions ... hoto-album -> 92 + 92 + 92 + 92 = 92%

- Extension with 2 x 100% rating on each review, then overall rating = 96% or just 76% ?!
http://extensions.joomla.org/extensions ... ng-content -> 96 + 96 + 96 + 96 = 76%
http://extensions.joomla.org/extensions ... tube-video -> 96 + 96 + 96 + 96 = 96%

- Extension with 3 x 100% rating on each review, then overall rating = 97% or just 76% ?!
http://extensions.joomla.org/extensions ... cookie-bar -> 96 + 96 + 96 + 96 = 76%
http://extensions.joomla.org/extensions ... for-joomla -> 97 + 97 + 97 + 97 = 97%

- And let's check other extensions on JED, you will see many other extensions get the same issue.

And you can see that seems all extensions that have lower overall rating are Free extensions, it seems some thing wrong with criteria "Value for money"?!
http://www.joomultra.com - Premium Joomla Extensions

deleted user

Re: Wrong overall rating score

Post by deleted user » Mon Jul 20, 2015 12:42 pm

I don't know if the scoring system behaviors are documented, but in some ways this is correct. The updated platform uses a system that depreciates the value of a review as time passes, so reviews that are 6-12 months old, even if they were all 100% scores, don't count as a 100% score any longer. Part of the argument is that a review made on an older version of the extension may not apply the same way if it were written today (code has been updated, features added/removed, etc.). So while those are still good reviews, they have a bit less value to the product today than the time they were written since conceivably the product has changed some since it was written.

qvsoft
Joomla! Apprentice
Joomla! Apprentice
Posts: 27
Joined: Wed Apr 15, 2009 7:22 am

Re: Wrong overall rating score

Post by qvsoft » Mon Jul 20, 2015 1:10 pm

I'm beleive that the score calculation is incorrected.
See my first example:
http://extensions.joomla.org/extensions ... no-doubles
http://extensions.joomla.org/extensions ... hoto-album

One submitted since 01 Jul, one from 28 June
And the review, one since 04 Jul, one from 03 Jul
But the overall score is very different.

And you can check all other new free extensions, you will see all of them get first 100% rating will have only overall 72%, but paid extensions will have 92%

http://extensions.joomla.org/browse/new
http://www.joomultra.com - Premium Joomla Extensions

deleted user

Re: Wrong overall rating score

Post by deleted user » Mon Jul 20, 2015 2:53 pm

I'd say keep following the JED's JIRA tracker on this one. Seems like it's a known quirk still. But, as far as I'm aware, it is correct that reviews do lose "value" over time. But newer items definitely shouldn't be seeing this decrease.

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Tue Jul 21, 2015 9:57 am

Michael, I'm seeing a similar frustrating trend. I've held off commenting on this because I didn't want to come across as a whining ninny, and wanted to give the system time to settle down with new scores to establish the behaviour.

Sorry for the length of the post, but it's been brewing for a few months now.

Look at Booking & Reservations

All of these components are currently listed above Jomres.

Rentalot plus
4 reviews on the current JED build, 1 last September then the previous 1 was a year before.

Solidres
1 review on current build. 2 older JED version but < 1 year.

Vik Restaurants
1 recently, 1 older JED version but < 1 year.

JMultipleHotelReservation
0 recently, 1 older JED version but < 1 year.

Koparent
2 recently, 0 older JED version but < 1 year.

Vik Rent Items
1 recently, 2 older JED version but < 1 year.

To add insult to injury, Jomholiday, who were kicked off the JED for a month a couple of months back for fake reviews are back on now and in the space of a few weeks have gained 6 shiny new and decidedly fake looking reviews. And guess what, they're scoring better than Jomres too.

Compare these to Jomres

Jomres
16 recently, 5 older JED version but < 1 year.

At this rate I'm going to get pushed off the first page completely soon, then business will become impossible*

So, total number of reviews in the last year

Rentalot 5
Solidres 3
Vik Restaurants 2
JMultipleHotelReservation 1
Koparent 2
Vik Rent Items 3
Jomres 21

The current scoring mechanism is punishing me for gaining more reviews, (currently position 11 of 15 and dropping monthly ) whilst those components with fewer but good reviews are best served by not soliciting reviews at all. This is contrary to what we want all want for the JED, which is lots of recent, relevant reviews which accurately reflect the constantly evolving Joomla ecosphere.
The updated platform uses a system that depreciates the value of a review as time passes, so reviews that are 6-12 months old, even if they were all 100% scores, don't count as a 100% score any longer.
As you know, I've always been fully behind this, I think it's an excellent method of judging a component based on it's recent activity. What I think's happening is two-fold :

The depreciation of review values isn't strong enough, if it's happening at all.

For example JMultipleHotelReservation. It's 6 places above Jomres on the page, scoring 100 points, but has only received one review in the entire last year. Their forum hasn't received any posts so far this year, and in 2014 there were only 5 topics raised. Compare that to the Jomres forum where there have been 27 active threads in the last 24 hours alone!

Another example, Vik Restaurants, with just two reviews in the last year is scoring better than Jomres. Ignoring the fact that they're not targeting the same marketplace, they've received just 6% of the total number of reviews compared Jomres, in the last year just 10% of the number of reviews over the same period. Either their review values aren't being depreciated at all, or the single recent review is hammering them up to 100 points. This is something that a huge, complex system ( Jomres ) with thousands of users simply can't ever achieve. This leads me to....

The number of reviews received recently isn't being taken into account at all.

I'm pretty sure I remember that this was going to be used in the algorithm in previous discussions, and as you can see from the previous point I made, it's an important one. The two are linked.

There are three people working full time on Jomres. We're pumping out new versions typically monthly. Every significant version contains new features that people have requested, but all of this work is only hurting us in the longer term. We'd be better off soliciting only excellent reviews from a select handful of happy customers and leaving the older reviews to fall off the end of the map. That's not good for the JED or Joomla, but it's the only practical solution to the problem that we're experiencing at the moment.


P.S. I'm not complaining about any of my competitors listed above, there are several that, whilst not as sophisticated as my offering, are still great at what they do. This post is about the scoring system only.

* Yes, I know that we're not supposed to use the JED as our sole source of traffic, but with the best will in the world, that's not possible. JED accounts for at least 50% of my traffic, always has and always will ( well, not if the current scoring arrangement continues). Like it or not, top of the first page in your category is the only place to be.


[edit for formatting and links]
Last edited by Vimes on Tue Jul 21, 2015 4:27 pm, edited 2 times in total.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Tue Jul 21, 2015 11:59 am

Further proof the scoring weighting is broken :

Free to book button

http://extensions.joomla.org/extensions ... ing-button

Hasn't received a review since 10 April 2013 ( which is > 2 years ago ) is still scoring 96 points, only one point below Jomres.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

rrmdp
Joomla! Apprentice
Joomla! Apprentice
Posts: 8
Joined: Tue Sep 22, 2009 2:53 pm

Re: Wrong overall rating score

Post by rrmdp » Tue Jul 21, 2015 2:51 pm

Been having a close look into the JED behaviour on Vince's post and I think he is 100% right.

Please JED dev's what do you think about his post ???

User avatar
jk1
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 213
Joined: Thu Sep 21, 2006 8:53 pm

Re: Wrong overall rating score

Post by jk1 » Tue Jul 21, 2015 5:41 pm

rrmdp wrote: Please JED dev's what do you think about his post ???
I think the JED team is aware of the issue(s):
https://joomlaextensionsdirectory.atlas ... e/JDEV-945

JazParkyn
Joomla! Apprentice
Joomla! Apprentice
Posts: 9
Joined: Tue Dec 17, 2013 10:21 pm

Re: Wrong overall rating score

Post by JazParkyn » Tue Jul 21, 2015 7:17 pm

The JED team is aware of this issue. More information can be found from the JIRA issue tracker:

- https://joomlaextensionsdirectory.atlas ... e/JDEV-945
- https://joomlaextensionsdirectory.atlas ... e/JDEV-873

If you believe that an extension has false reviews, please report the review so that the team can deal with it through the proper channels - support cannot be provided through the forums.

If you believe that your extension's score has been incorrectly calculated, please do not open excessive amounts of tickets - the team is aware of the issue. The above links give more insight to the calculations and weighting of reviews.

"Scores take a number of factors into account.

If an extension has less than 5 reviews, the score is penalized.
If an extension has more than 5 reviews, a few more points are added to the score.
A percentage of reviews are taken off of the top and the bottom to account of anomalies.
When a review is submitted, it isn't factored into the score for 7 days.
After the 7 day period, recent reviews impact a score, but depreciate in value over a two year period." - (Matt Baylor, https://joomlaextensionsdirectory.atlas ... e/JDEV-945)

Kind Regards
Jaz Parkyn
JED Listing Specialist

rrmdp
Joomla! Apprentice
Joomla! Apprentice
Posts: 8
Joined: Tue Sep 22, 2009 2:53 pm

Re: Wrong overall rating score

Post by rrmdp » Thu Jul 23, 2015 7:52 am

Thanks @JazParkyn, all I know right now is that Jomres is the more mature and probably the best booking engine Joomla based and is almost going to 2nd page of "Booking & Reservations" http://extensions.joomla.org/category/v ... servations as you can see in le link or this screeny http://take.ms/0H6Vg Hope this is fixed soon.

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Thu Jul 23, 2015 8:28 am

The JED team is aware of this issue. More information can be found from the JIRA issue tracker:
Neither of those bug reports refer to the problem in question Jaz, they simply note that the scoring's incorrect.

This one does

https://joomlaextensionsdirectory.atlas ... e/JDEV-991

If you believe that an extension has false reviews, please report the review so that the team can deal with it through the proper channels - support cannot be provided through the forums.
It's already been reported by somebody else.
If an extension has less than 5 reviews, the score is penalized.
If an extension has more than 5 reviews, a few more points are added to the score.
A percentage of reviews are taken off of the top and the bottom to account of anomalies.
When a review is submitted, it isn't factored into the score for 7 days.
After the 7 day period, recent reviews impact a score, but depreciate in value over a two year period." - (Matt Baylor, https://joomlaextensionsdirectory.atlas ... e/JDEV-945 )
I'm going to have to pull out my "old man" card here.

Please don't assume I'm an idiot Jaz, I've been using and thinking about the JED probably since before you were aware that Joomla existed. It's scoring system(s) have been a huge bone of contention for me.
If an extension has less than 5 reviews, the score is penalized.
When this was discussed, back in the day, it was believed that reviews over 2 years old would be discounted. This made sense at the time, and if this were the case now, several of the extensions on the bookings and reservations page would no longer be considered viable options for users to investigate. As it stands at the moment, the scoring system is pushing just extensions with old but good reviews to the top of the list, making the review system pretty much worthless for anybody wanting to find a quality extension.
A percentage of reviews are taken off of the top and the bottom to account of anomalies.
This doesn't seem to be working for extensions that have just a handful of sparkling reviews. Rentalot has had just 2 reviews in the last 4 years yet it's still scoring 96.
After the 7 day period, recent reviews impact a score, but depreciate in value over a two year period." - (Matt Baylor, https://joomlaextensionsdirectory.atlas ... e/JDEV-945 )
See my comment above, the depreciation isn't working ( effectively ). Freetobook, last reviews were > 2 years ago. It still scores 96. It's at v1.0.0 so it's likely that the module was added sometime before the first review in 2011, so it hasn't been updated since. It may well still work, but it's just a simple button to a SaaS service so it's not really adding much value to Joomla itself. That probably explains the lack of reviews.

On the other hand, you've got a system like mine, which has been a loyal supporter of Joomla for 10 years. I've always played by the rules, even when I knew that the competition was cheating and the old JED Team were powerless to act against them.

It's the same on http://extensions.joomla.org/category/vertical-markets. Jomres doesn't even appear on the first page any more.

Let me make this clear. The JED's scoring system is flawed. Developers like me, who believe that the path to restoring Joomla to it's former glory by having a vibrant extension market are being actively damaged by the scoring system. It's getting to the point that sales have dropped so far that I'm not sure that I can maintain business for much longer.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

User avatar
nonumber
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 223
Joined: Mon Sep 18, 2006 1:17 pm
Contact:

Re: Wrong overall rating score

Post by nonumber » Thu Jul 23, 2015 10:30 am

The technical documentation on how the JED rating system should work is fine.
It has been a work in progress for many years.

The implementation, however is very bad and very broken.
It was already broken in the JED 3 alpha/beta stage and still is.

The current setup for it (or at least, how it should be) is based on ideas we (I) had for it some years ago (when I was still on the JED team). I even wrote the code for it to implement in the old JED. But it never got implemented, seeing efforts went into writing up the specs for JED 3.

I have offered to rewrite the whole rating section of JED 3 to fix this. And also get my hands on the graphical side (getting rid of these terrible circles!).
But as far as I know the JED 3 project still hasn't been handed over as finished.

My offer still stands. So, JED team: as soon as the JED 3 project has been delivered and handed over to the JED team, I am willing to rewrite what is necessary to make it workable.

The idea was also to have a final rating (score, rank, whatever) that is not necessarily a number to 100.
Now you still have many listings with a rating of 100, which results in the same issues as we had before. A listing with 15 (and now even with 1) rating of 100 being above a listing with 200 reviews, but a score of 99.
My idea would be to have a more unique ranking/scoring number. So not out of 100, but maybe with a max of 5000. Or even a completely unique ranking number like Alexa does.
Ranking should be based on the stuff already there in the JED 3 specs, like number, age and value of review, 'normalizing' by ignoring top/bottom x%, etc.
Maybe even have the registration age and total number of reviews of the reviewer play a role.
NoNumber.nl - Extensions for Joomla!
Advanced Module Manager, Articles Anywhere, Better Preview, CDN for Joomla!, Cache Cleaner, Components Anywhere, DB Replacer, Modals, Modules Anywhere, ReReplacer, Sliders, Sourcerer, Tabs, Tooltips, What? Nothing!

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Thu Jul 23, 2015 1:13 pm

More evidence that the scoring is in need of a good shake up.
If an extension has less than 5 reviews, the score is penalized.
What's the penalty? A cuddle?


Team Display
2 reviews score 96

JUX 3D Restaurant Menu
2 reviews score 96

JomEstate PRO
2 reviews score 96

Freetobook Booking
4 reviews, last 10 April 2013. score 96

Navega Bem - Real Estate
4 reviews score 96

rentalot
2 reviews in the last 4 years. Score 96

JHotelReservation
No reviews in the last year. Score 97

Hotel Directory
Two reviews in the last year. Score 98.

These aren't edge cases, or one-offs. These are the norm.

I'm 100% behind nonumber's suggestion that he be allowed to experiment with refactoring the scoring mechanism so that it's fairer to all. Count me in if help is needed.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

rrmdp
Joomla! Apprentice
Joomla! Apprentice
Posts: 8
Joined: Tue Sep 22, 2009 2:53 pm

Re: Wrong overall rating score

Post by rrmdp » Thu Jul 23, 2015 7:42 pm

Glad to see Peter interested in one of the most important Joomla's features the JED, and IMHO with out a fair and good rating algorithm it looses its most important sense... the only way to keep and get better extension for the usrers.

qvsoft
Joomla! Apprentice
Joomla! Apprentice
Posts: 27
Joined: Wed Apr 15, 2009 7:22 am

Re: Wrong overall rating score

Post by qvsoft » Tue Aug 11, 2015 3:38 am

Any one from JED team can let us know when this issue will be fixed?
This is an important issue, and just need to change the score formula and rebuild the rating, but it takes many months and still not get fixed.
http://www.joomultra.com - Premium Joomla Extensions

User avatar
Claudia_A
Joomla! Explorer
Joomla! Explorer
Posts: 402
Joined: Thu Jul 17, 2014 6:36 am
Location: Bucharest
Contact:

Re: Wrong overall rating score

Post by Claudia_A » Tue Aug 11, 2015 1:57 pm

mbabker wrote:I don't know if the scoring system behaviors are documented, but in some ways this is correct. The updated platform uses a system that depreciates the value of a review as time passes, so reviews that are 6-12 months old, even if they were all 100% scores, don't count as a 100% score any longer. Part of the argument is that a review made on an older version of the extension may not apply the same way if it were written today (code has been updated, features added/removed, etc.). So while those are still good reviews, they have a bit less value to the product today than the time they were written since conceivably the product has changed some since it was written.
Shouldn't this be actually the opposite way? I mean an older version is much stable, then a new released one, which in time if any issues, the developer can fix them immediately as signaled.
Best Regards,
Claudia A.

http://www.thePHPfactory.com || Professional Joomla! Extensions

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Tue Aug 11, 2015 2:04 pm

Claudia_A wrote:
mbabker wrote:so reviews that are 6-12 months old, even if they were all 100% scores, don't count as a 100% score any longer.
Shouldn't this be actually the opposite way? I mean an older version is much stable, then a new released one, which in time if any issues, the developer can fix them immediately as signaled.
IMO it's a good strategy, so long as the older reviews still count for something (less extreme than what we're seeing now), and the final score reflects the number of older reviews too.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

User avatar
nonumber
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 223
Joined: Mon Sep 18, 2006 1:17 pm
Contact:

Re: Wrong overall rating score

Post by nonumber » Tue Aug 11, 2015 7:10 pm

NoNumber.nl - Extensions for Joomla!
Advanced Module Manager, Articles Anywhere, Better Preview, CDN for Joomla!, Cache Cleaner, Components Anywhere, DB Replacer, Modals, Modules Anywhere, ReReplacer, Sliders, Sourcerer, Tabs, Tooltips, What? Nothing!

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Mon Sep 28, 2015 12:33 pm

Is it any wonder that developers are abandoning Joomla in their droves?
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

User avatar
nonumber
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 223
Joined: Mon Sep 18, 2006 1:17 pm
Contact:

Re: Wrong overall rating score

Post by nonumber » Mon Sep 28, 2015 12:47 pm

It seems that getting stuff listed on the WordPress directory is no fun either.
NoNumber.nl - Extensions for Joomla!
Advanced Module Manager, Articles Anywhere, Better Preview, CDN for Joomla!, Cache Cleaner, Components Anywhere, DB Replacer, Modals, Modules Anywhere, ReReplacer, Sliders, Sourcerer, Tabs, Tooltips, What? Nothing!

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Mon Sep 28, 2015 4:54 pm

I found it really easy, tbh. Now, Wordpress users, that's another thing altogether.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

rrmdp
Joomla! Apprentice
Joomla! Apprentice
Posts: 8
Joined: Tue Sep 22, 2009 2:53 pm

Re: Wrong overall rating score

Post by rrmdp » Fri Oct 21, 2016 9:40 am

More than a year since the last post and the JED rating is still the same :(

User avatar
Vimes
Joomla! Ace
Joomla! Ace
Posts: 1675
Joined: Fri Aug 19, 2005 12:14 am
Location: United Kingdom
Contact:

Re: Wrong overall rating score

Post by Vimes » Fri Oct 21, 2016 10:45 am

Yeah, there's something hugely wrong with the scoring, still. I just checked

https://extensions.joomla.org/category/ ... -directory

and Team Display is still scoring 96, 15 months later. It hasn't had a review in 28 months, and still only has 2 reviews.

I'm not attacking TD, just using it as an example to demonstrate the scoring behaviour.
http://www.jomres.net THE online hotel booking and reservation system for Joomla and Wordpress.

HelpJ
Joomla! Intern
Joomla! Intern
Posts: 80
Joined: Thu Dec 01, 2016 3:38 pm
Contact:

Re: Wrong overall rating score

Post by HelpJ » Tue Dec 06, 2016 12:16 pm

A year and a half passed and nothing changes? JED wake up.

User avatar
nordmograph
Joomla! Enthusiast
Joomla! Enthusiast
Posts: 120
Joined: Thu Jan 28, 2010 12:38 am
Location: Lille France EU
Contact:

Re: Wrong overall rating score

Post by nordmograph » Thu Jun 15, 2017 9:34 am

Maybe we should give more weight to this issue on the issue tracker
https://joomlaextensionsdirectory.atlas ... ?filter=-5
E-commerce, community & geo mapping extensions
https://www.nordmograph.com/extensions

User avatar
stutteringp0et
Joomla! Ace
Joomla! Ace
Posts: 1389
Joined: Sat Oct 28, 2006 11:16 pm
Location: Texas
Contact:

Re: Wrong overall rating score

Post by stutteringp0et » Sun Jun 10, 2018 4:05 am

It's sad to watch the issue/forum post go nowhere. I'm in the same boat as Vimes - one of my extensions has more reviews than the top 10 combined - more 100% reviews than the top 5 combined, yet I'm listed below many 1 review extensions. It's frustrating.

It seems like every time I get a new review, even if it's 100% - my score falls and so does my position in the list. Soon I won't be on the first page of results, even though I spent the past 6 years at #1.
My extensions: http://extensions.joomla.org/profile/pr ... ails/18398
Honk if this signature offends you.


Locked

Return to “extensions.joomla.org - Feedback/Information”