JED Rating System Evaluation

[masterchief]

A few of us (me included) started to wander off topic in this thread:
http://forum.joomla.org/viewtopic.php?f=262&t=544219
where the question of what to do with old rating and reviews was put. This thread is to split out the discussion of alternatives or improvements to the rating system on the JED so we can keep the other thread clean

The problem with the 1-5 rating system as I see it is that it's a very subjective scale where each value is not equal for all. One person's 4 is another's 5, and so on. Coupled with that is what does a 1 mean and what does a 5 mean. Is 1-5 really bad to really good or is 1-5 a range of "it installs" through to "it does everything I want it to". All up this presents confusion to visitors trying to interpret what a rating of 3.64 means, and it also sends a confusing message to the developer.

My suggestion is to ditch the current rating system and replace it with a single "I use this" system similar to http://osx.iusethis.com You would retain the favourite system but a registered user would simply state whether they use an extension or not. From time to time, they would go through their list and update what they aren't using anymore, and catch up on things they realise they've been using for a while. The big advantage to this is it's not a subjective rating - it's either a yes or know and easily interpreted visitor and lister alike. It also avoids the negativity of the protest 1 and 2 voting that you can get. Either a person uses it, or they don't. If it's used and favoured, people can draw their own conclusions.

Such a system would allow for useful badges a lister could use on their site, eg "123 people use this" (compare that to "3.64 rating out of 5 votes on the JED"). I would also suggest change the ordering on the listing pages to be via either Most Used, Most Favoured or Last Updated. That gives 3 good, objective (relatively anyway) metrics to help a visitor choose an extension. It would also allow for pages for individual users like this:
http://osx.iusethis.com/user/eddieajau

I think pages where you can see what other people are actually using would be a useful tool for users of the JED, certainly more useful than just trying to interpret what the average of a 1-5 rating means. What we are after is a way to help the user triage the vast number of extensions available. Once they have a shortlist, obviously the reviews come into play. At least, that's how I use the I Use This site.

Anyway, that's a sketch of an idea - not perfect by any means. Over to others to throw other ideas into the melting pot...

[Alex Dobrin]

I think this system would be a much better voting system then the current one.
It is removing the problem with voters that are not submitting a review to explain their vote and the intervention of a moderator is no longer necessary when somebody wants to change the vote.
I would just add one more way to ordering the extensions : "Most reviewed" or maybe "Most reviewed in the last 12 months" to consider the discussion on the other topic.

[ot2sen]

Really like this idea Andrew

In a clean and simple way it removes all the doubt and misunderstanding that can easily be attached to especially the 1-3 votes. It is rather difficult for other users to verify the reason of those votes and the possible connected negative reviews, as they could relate to hosting, user errors or in fact to the extension in target.

Seems much better with a strong number of 'I use it'. Good idea!

[vdrover]

I personally like the 1-5 rating. Rather than ditch it entirely, we have been thinking of a more detailed rating system where various aspects of the extension can be evaluated independently. This can be coupled with a requirement that all votes require a written review.

As far as "I use it", it seems like this is a variant of the "favored". Can you explain how they are different?

[mlipscomb]

What we are after is a way to help the user triage the vast number of extensions available.

I believe that this is one of the most important points. If users feel comfortable finding and selecting extensions then it provides benefit for the developers as well. It becomes a "community roundtable" about the various extensions rather than a subjective 1-5.

I do see value in a rating system, but not in the current one we have. I would like to see ratings be the following starred votes:
Overall Rating
Is the extension easy to use?
Was the extension easy to install?
Overall Rating of Support if used?
Overall Value?

Combine that with the "I use this", and weight reviews with the new "This review was helpful" count (i.e., most helpful reviews at the top, and then date descending) and you have a much more equaled view of the real extension.

This provides for a much more "personalized" experience on the JED.

Additionally, we can easily add in the social sharing buttons that are found in the new JUG directory, which could also add weight to a listing - for example - if 500 people "Like" the extension on FB or Tweet it.

If a ranking system combines all the above, it becomes much more practical.

[Alex Dobrin]

I think it's important to keep the rating system as simple as possible.
Adding more things to rate on an extension will complicate things so much that fewer and fewer people will submit a complete vote or will make people just chose a random vote on the parts they didn't test or are not interested.
And considering the subjectivity of a 5 start vote the end results will be in many cases completely different from the reality of the extension.

[lafrance]

I think it's important to keep the rating system as simple as possible.

I do not think it will it would surely drop fake vote. And users that really like the extension will not mind doing it.
The rating would be much fair for the users as it would be for the developers

[mustaq-ahmed]

@ masterchief

These are very good suggestions and well thought of obviously.

With any type of voting/review system we should look at ways of lowering the barriers for Users to interact with the JED, so I am also for the " Keep it Simple " principles.

@ mlipscomb

Regarding " Is the extension easy to use? " and " Was the extension easy to install? " I believe that users will interpret these differently based on their Personal level of experience with Joomla! and 3rd Party Extensions. There are many examples of extensions that work brilliantly/flawlessly but that may require some "above average"user experience to configure ( not modify/customise ) , resulting in the extension receiving a poor rating for these specific metrics. I am basing this observation from experience with helping clients who all but gave up on properly configuring an extension. We must take into account the User experience Level of the majority of users visiting the JED.

[mlipscomb]

@mustaq-ahmed:

That's a great point - maybe included "Experience w/ Joomla - X Years" would bring a more well-rounded view to that. Possibly to even show an overall rating of users with X amount of years experience.

[Alex Dobrin]

I do not think it will it would surely drop fake vote.

How ? A fake voter will have no problems making 5 clicks instead of one.

And users that really like the extension will not mind doing it.

I'm sorry, but I think the exact opposite.
Some will mind and will either not vote or vote randomly.

That's a great point - maybe included "Experience w/ Joomla - X Years" would bring a more well-rounded view to that. Possibly to even show an overall rating of users with X amount of years experience.

Wouldn't that complicate things even more ?

[vdrover]

How ? A fake voter will have no problems making 5 clicks instead of one.

Currently, drive-by voting (anonymous voting without a written review) is trivial and common.

If voting is restricted to registered users, a few extra categories added to the ratings, and all votes accompanied by a written review (say 25 word minimum for example), drive-by voting will be eliminated entirely and editors can compare a detailed vote to a written review. This will not only discourage fraud (more work for user) but provide a better review.

[masterchief]

As far as "I use it", it seems like this is a variant of the "favored". Can you explain how they are different?

Yes, it's the same a favoured. It can probably mean different things to different people but that way I use it on iUseThis is there are things I use because it's there, and there are things that are like my right arm. An example in the Joomla world is I could "use" many extensions when I'm developing sites for people, but I don't necessarily like all of them. Docman I use, but don't like it. Sam's update manager I use and I love it.

So I would leave the existing favoured button and replace the 1-5 rating system with "I Use This".

On the topic of extended ratings, again I think you just have the same problem spread over a wider area. That, in my opinion, is what you use a review for to put into your own words what you think.

[horus_68]

My proposal its to change everything

No more reviews and no more votes... well not really a "no more":
More likes, less text, better texts, less votes, better votes

Actually the number of votes its very low and almost no extension has +100 votes! The reviews are also very few and almost of no use for users or developers!
So lets change this.

Proposal
1- Improve the JED MyPage (into a JED social network)
- Rename the actual Favored MyPage to: Extensions that I use now (limited to 40 - no self extensions)
- Ability to filter extensions favored by update date, name, category
- Create a page: Extensions bookmarked for later analysis (limited to 40 - no self extensions)
- Create a new page: My top 3 personal extensions - for recommendations to users (only self extensions!!)
- Create a new page: My top ten picks - for recommendations to users (only one for each top category, no self extensions)
- Page: My developer friends (other DEVs connections)
- Page: Fan page (users that follow developer)
- Page: Latest updates from this developer
- Improve profile infos: this is my sites, this is my experience on Joomla, This is my PeopleJoomla page, I'm also listed at JRD and forum, etc.

This would be visible to the public in a single good looking page (this would be the user page linked to the profile of user and developer). It will include a contact link or form.


2- Improve Reviews:
- Reviewer profile visible (by user choice, as he may or not want to be contacted by Devs)
- Reviews and votes will be the casted at same time
- Reviews are time based but not removed: so users can edit their review in a 3 months period (this will change the review status as a new review with new date)

- "Was this review helpful to you? Yes No" [already implemented]
-- Review highlighted by users: The review considered most "useful" by users votes will get a special spot on the list
-- Review highlighted by developer: Select a review as most "useful" to get a special spot on the list
-- Filtering reviews by order, date, "useful", votes on extension

- The review form:
-- Create a multiform (thematic box) for different aspects of extension analysis
-- Limit the amount of characters on each box (140 char... no URL !) so its easy to review and to read by others
-- User need to write at least in one box
-- Only the boxes with words will be visible when review published

Areas to review
- Easy to use - frontend
- Easy to use - administrator (settings/management)
- Easy to install and update
- Design and Looks
- Direct support (forum, email)
- Docs and Guides (Faq, articles, manuals)
- Technical quality and security (Code)
- Ability to customize / extend

Note: Direct support and docs are both support but splitting will be good to free extensions that usually don't have top direct support but can have good documentation.

3- Improve Votes
- Votes with reviews only (at least one box filled)
- Votes can be updated each 3 months by the user (time based but not removed. editing votes will update the date of the review also). This will reflect more accurately the actual extension and not the old extension. Also developers could also feel more interest in updating their extensions
- Less rules limiting the vote requesting by developers. No paid campaigns but no restrictions on the way to promote voting
- Developers would be blocked to vote on any extension at all
- Votes in Pie Chart:
Users get a total of 20 points to split over 8 areas of the extension, so this should reveal what area of the extension has more importance for each user.

Note: each votes will have always a total of 20 points, but users only are required to vote in 3 areas so

Areas to vote
- Easy to use - frontend
- Easy to use - administrator (settings/management)
- Easy to install and update
- Design and Looks
- Direct support (forum, email)
- Docs and Guides (Faq, articles, manuals)
- Technical quality and security (Code)
- Ability to customize / extend

This "Pie Chart vote" would remove the actual in-utility of votes (5 stars don't say nothing at all) and bring to zero the number of reports from developers (there is no bad votes... as lots of DEVs actually report any 4 stars!).
Also: each user is different, and each developer could see then what kind of user is attracted by their extensions

4- Like this extension
- Rename the "favored" to "like" to be closer to other social platforms
- See #1 on Mypage enhanced for "extensions that I like"

5- Categories and ordering
Actually this is the big problem at JED, for users and for developers... so:
- Ranking system (weighed votes and rating algorithm)... lots of items to consider for this algorithm as stated in #1, #2 and #3 !!
- Rating algorithm should also consider the date of all elements included: votes, favored, picks
- Personal filters when browsing JED (date updated, votes, free/commercial, name)

If more votes, more items to rank and date included, then:
less fraudulent actions,
less work to monitor and review approval
more quality on informations
better organization

[matt_nz]

Some thoughts on the 'I use this' as a replacement to voting.

Good Point:
It makes each user's input (a single vote or a single addition to the 'I use this total') more reflective of a single input.

On extension directory categories where there are lots of extensions all crammed around the 4 stars mark, one bad vote may be enough to drop an extension down 10 places, and effectively half the amount of traffic to the site.

This makes incentives for false voting, and incentives for complaints about suspect voting, and ends up taking JED editor's time. In a fair system, one user should not be able to effect the extension's place in the listings to this degree.

Regarding this point, the 'I use this' would be great as an alternative. If 200 registered users say they use one extension, and only 150 say they use another, this usage is a good sign of how much people like an extension, bad extensions don't get used much.

If one person removes their 'I use this', it would have a measured effect (200 to 199), not a 5th place to 15th place on the directory.

Bad Point:
Free extensions get used more than commercial ones, in virtue of them being free, and if the extensions are listed according to most used, commercial developers will have a huge disadvantage.
Especially commercial developers going towards the high price, high product and support market. It may create an incentive for commercial developers to avoid this market.

[edo888]

I like Andrew's proposal.

Another idea is to have reviews/votes to have expiration time. And the ordering algorithm changed to value 5s in the first place and 1s in the last.

I would like to see also a developers "rating" page in the JED. It will help to increase the spirit of competition between developers and will bring new extensions!

[brian]

I really like the idea of an "I use this button" especially if there was a way of seeing everything that someone "uses".

[skOre]

Pah, stumbled over this a little too late. I have discussed abolishing the 5-star review with some other folks before, so I'm a bit late to this particular party, but it has been on my mind for months now.

The most important point is not finding what is 100% fair, but what is most useful.

The 5-star system is already an attempt at making it as fair as possible and all the ventures into "let's just split up the vote to multiple questions" fall for the same fallacy. There is, of course, no perfectly fair system and it's also not what people need. They need to know whether something will solve their problem. An "I use this" would indeed be the best of both worlds here. The crucial thing, however, is to encourage people to build a "portfolio" where they present what they use and what they use it for.

It's much more helpful to read a portfolio of somebody who, say, mostly builds a lot of Online Magazine websites and describes her process when you're looking for the right extensions to build one yourself.

I know that this is already somewhat there in the current system, but I think if the JED emphasized that you can express yourself in the portfolio you put up, you will get information that is a lot more useful than many disjointed reviews can transmit.

@matt_nz - I agree that this would skew the favor for components that are easier to access, like the ones that are free or don't require a registration to download. However, I think that this is actually a benefit and creates a more healthy kind of competition when compared to the 5-star rating.

[sourcecoast]

I stumbled over this late too, after skOre Tweeted about it.

I like the "I use this" concept as its easy and relatively unambiguous. My biggest issue with it is that it very, very, very heavily favors older extensions. Any extension released 2 years ago is going to have more "I use this" votes than one released last week (or last month, or likely even last month), guaranteed. Even if the one last week is the new hotness, by virtue of it being newer, it will sit at the bottom of the pile. That prevents new users from finding it and possibly using the old-standby that was coded for Joomla 1.0 and sort-of upgraded to function with 1.5.

I don't like the current 5 star voting scheme a helluva lot, but at least it takes 5 (I think) votes for an extension to get an overall rating. After that rating, it's compared against the remainder based on it's average.

There's really no way that I can see to do realistic, unfavored "I like this" rankings. The thought that users will come back constantly to update sounds great in theory, but even if they do, the oldest extension will still (most of the time) win.

Would love to hear thoughts on how to make it work though.

DEFINITELY agree with no review == no vote.

[Geoff]

I like the "I use this" concept as its easy and relatively unambiguous. My biggest issue with it is that it very, very, very heavily favors older extensions. Any extension released 2 years ago is going to have more "I use this" votes than one released last week (or last month, or likely even last month), guaranteed. Even if the one last week is the new hotness, by virtue of it being newer, it will sit at the bottom of the pile. That prevents new users from finding it and possibly using the old-standby that was coded for Joomla 1.0 and sort-of upgraded to function with 1.5.

I agree that this will greatly simplify our voting procedure. What we can do to overcome this "age" issue is probably use tiered rankings. Instead of listing extensions just by most "I use this", we can break it down to most "I use this" by day (for current day), week, month, year, and then all time. Each section/category would be sorted by default to list extensions with the most "I use this" by week. Users can then choose to resort the section/category by most "I use this" day, week, month, etc.

As for what sk0re is talking about, the general idea was brought up a while back (but faded) away. The idea was to have a section like Add-On collections for firefox: https://addons.mozilla.org/en-US/firefox/collections/

[matt_nz]

I think the issues are around commercial vs non-commercial need more discussion.

My experience of site traffic is that being number one in a category leads to at least 5 times as much traffic as being number 20. For commercial developers, there is a huge incentive to get near the top.

If the above system was introduced, there would be a big incentive to make extensions that are small, easy to use, lower price, and widely adopted. These ones would get used a lot, so they would get a lot of traffic.

If someone decides to make a big project (for example magento bridge), that is known to need a lot of support, and is given a high price because of this, it is likely the big price will mean it is only adopted by big e-commerce sites. The smaller Paypal cart that costs ten times less may get much more widely used.

Because of this, commercial developers might think 'Do I want to take on a big project like a magento bridge, if only the far end of the bell curve will use it, and I will sit at the bottom of the 'I use this' rankings. Wouldn't it be better for me to make a small Paypal cart that gets to number 1.'

If the 'I use this' system ends up giving incentives for commercial developers to make lots of small lower priced software, then this could be a bad move.

[lafrance]

I think the issues are around commercial vs non-commercial need more discussion.
Because of this, commercial developers might think 'Do I want to take on a big project like a magento bridge, if only the far end of the bell curve will use it, and I will sit at the bottom of the 'I use this' rankings. Wouldn't it be better for me to make a small Paypal cart that gets to number 1.'

If the 'I use this' system ends up giving incentives for commercial developers to make lots of small lower priced software, then this could be a bad move.

Maybe then the solution could be 2 sections commercials and free?

However free extension will always be more popular as it free,to give higher rating to commercial because it took tons of hours to code should not have an extra voting as it would not be fair for free extensions developer thinking well if commercial get higher rating because you pay then you get into problems.

I think a better solution is required hope someone can think of one.

This one do not think would work

Pierre

[mark_up]

I have to agree with matt_nz.

At this point Ninjaboard would have only a tiny fraction of Kunena users (given it's 5yr, million+ download history)

Who would even dare create a backup component given Akeeba's years of dominance in that area?

Would Azrul have invested in JomSocial given it would be impossible to get a decent position on the JED at the prices he was charging for it?

Wordpress must be better than Joomla, given it has millions more users, right? (hint: if you tell me WP is only better if you're building a blog, you'll just make my point)

[sourcecoast]

Didn't think about the commercial vs free aspect, but it basically comes down to the same argument that I made - older (or free) extensions are heavily biased towards in an "I use this" environment. Even if you do it over a date range (week, month, year), the free/older extension will likely win.

The star system, while flawed for ambiguity sake, can easily be averaged which levels the playing field IF everyone used the same system.

To continue using the star approach, I'd like either the more granular system mentioned above (Installation, general use, etc) OR I'd propose a 3 star system. 3 stars, to me, is pretty obvious. Think about what you'd consider each star meant if you want before you read my description below.

1 star - doesn't work / doesn't do what it says (Poor)
2 stars - works. Maybe not perfect, or has some rough edges (Average)
3 stars - works great, does everything it says (Excellent)

With 5 stars, 2 and 4 stars always seem very ambiguous. Of course, with even 3 stars, I'd recommend giving the guidance above (Poor, Average, and Excellent)

[masterchief]

I think the Apple App store deals with the free vs commercial very well. I will say though, you don't want to give one weight over the other. You need to treat both with the same rules.

[skOre]

Let's not get carried away here. The goal, as I've said it before, is not to be 100% fair, but to give the best result to people looking for an answer.

So if there is Akeeba and everybody uses it - that is a message that can be represented in it holding a strong position, there is nothing wrong with that. If there is an underdog trying to do better, it will still be a fight, but that's not different to any other system.

I don't think that the commercial vs. free discussion needs a solution. To be perfectly honest - there are some software packages that require a business model and others that do not and as I wrote in my initial post: the evolution of the community will take care of this. You could just as well argue that commercial components have an easier time because while they don't get the same frequency that free ones do, it might allow the developer to live off his revenue from the software allowing for more time to make it perfect.

To sum up the rather awkward example: I think that we have enough factors even with the free vs. commercial to let the free market and a healthy competition do their job. If a software is destined to take the lead, it will happen.

As for the ongoing discussion about whether we should stick to X-star ratings:

I disagree. People don't look for poor, average or excellent extensions. They only care about such a ranking because it somewhat fits their greater goal: Finding the extension that is right for them. We should concentrate on making sure that that happens. Discussing which precise amount of artificial measuring we will apply will not help people build better sites.

Nobody is going to say: "Oh this software is rated excellent, but their support is rated average, that sure tells me..." nope, that doesn't tell anybody anything.

Just ask yourself what you do when you look for an extension:

• You look for the right search or category
• You single out a number of candidates
• You study the candidates in detail
• You go with the most likely candidate
• If it works out: Hooray! Otherwise: back to step three

What you're saying is: We could optimize the steps two and three of this by giving people less/more numbers. By that measure, I would never think, for example, that FlexiAccess is a good ACL component. It's ranked #8 in its category and has poor reviews. You know what does help me? Reading those reviews and seeing that every single one of them is by a person who couldn't even manage to download the software.

As an aside: Can somebody please clean that up? It's boggling my mind. Seriously.

What I want is to see "this many people found this software good" and then read their reviews and see why. That is what I can base a decision on.

I think the problem is that a lot of people think that the JED is supposed to give people an equal playing field in getting their extensions known. That is a misconception. Building software to solve peoples problems will get people to know about it. But that isn't something that is supposed to happen on an extensions directory.

Those two things out of the way, I think it should be clear why an "I use this" button plus a portfolio will be far more helpful.

[matt_nz]

@skOre

I think that we have enough factors even with the free vs. commercial to let the free market and a healthy competition do their job. If a software is destined to take the lead, it will happen.

I disagree. The way that extensions are ranked changes incentives for developers, therefore it is likely to shape the kind of extensions that get made. The rankings system is not a neutral free market where healthy competition will do it's job. It is a system that will inevitably have some biases, and if the bias is hugely towards the extensions in the middle of the bell curve, and against extensions at the end of the curve, then this will have a big effect on what extensions get made.

I think the problem is that a lot of people think that the JED is supposed to give people an equal playing field in getting their extensions known. That is a misconception. Building software to solve peoples problems will get people to know about it. But that isn't something that is supposed to happen on an extensions directory.

We should care about how the extension ratings effect developers, because what developers do effects what products are available to users.

[skOre]

@matt_nz

I cannot speak for everybody here, but I don't think that it works like this. People write software because there is a need for it, not because there is a financial or ranking incentive for it.

If ratings affect what you develop, then you should not develop.

[lafrance]

@matt_nz
I cannot see this done within jed it would start a war If an extension is commercial and popular it will be rated high also as long the price structure do not kill the average end users.
To make commercial the way you suggest it would discriminate then the next step would be developer asking for special reviews.

The playing field must be the same for all

Pierre

[matt_nz]

@skOre
It is a contradiction to say "I cannot speak for everybody" and then say "People write software because.."

By saying "People write.." you are speaking for everybody.

The gpl is about developers having the freedom for charging or not charging for what they distribute, and not to have others dictate to them how they should develop, or what business model they should take.

I appreciate you showing your thoughts about commercial development though, I think it helps the discussion if these views get out in the open.

@Pierre
"To make commercial the way you suggest" I'm not sure what suggestion you are referring to, can you please give some more info.
I don't think I have made any suggestion, just pointed out a bias in a ranking system.

[lafrance]

@matt_nz

Please give us a few days max as we should be able to bring some ideas that may solved some issues.

Pierre