An open proposal from the JED: archive old votes/reviews

[vdrover]

As you may know, we are continually striving to keep the JED modern and improve workflows for both end users, developers and JED editors. We’ll be consulting with the community more and more on these maters so that the JED works well and remains a valuable resource for all parties.

There are some changes however that can be done relative easily and quickly. One area that we have identified is the idea of ‘vote/review expiration’. Simply put, votes/reviews for an extension become less relevant as time goes on.

For example, many of the older votes/reviews on the JED today were based on the Joomla 1.0 version of an extension. In other cases, new features and many, many releases of an extension have made old votes/reviews inappropriate and do not represent the current state of an extension.

Thus, we would like the community input on an open proposal to archive any votes and reviews submitted to the JED prior to January 1st, 2008.

We have performed a preview of the changes from the Top 100 rated extensions and the changes in ratings are minimal. The changes in the Top 10 extensions are show below as an example:

Ranking, New Rating (Old Rating), New # of votes (old # of votes)

1. 5.000 (5.000), 34 (35)
2. 4.938 (4.895), 16 (19)
3. 4.938 (4.950), 16 (20)
4. 4.929 (4.933), 14 (15)
5. 4.923 (4.941), 13 (17)
6. 4.900 (4.902), 160 (163)
7. 4.895 (4.809), 57 (68)
8. 4.891 (4.831), 55 (65)
9. 4.880 (4.835), 482 (545)
10. 4.849 (4.796), 693 (802)

I think that last entry is very relevant: after archiving 109 votes, the change in the rating is +1.11%.

Feedback from all users of the JED are welcome.

[vdrover]

The average rating change for the Top 100 rated extensions is 0.0025/5 = +0.05%.

[NHRADeuce]

I like the idea of culling non-relevant reviews and votes. I don't think it should be tied to a specific date for all extensions though. Maybe for the initial cleaning clear out everything before 1/1/2008 but going forward it seems like it would make sense to archive reviews and votes based on when the extension was last updated.

For example, a component that is actively being developed with many new releases could be be earning significantly better/worse reviews and votes within a 3 or 4 month span.

Conversely, you could also have a basic plugin that has only had one release, that is compatible with 1.6 that hasn't changed nor will it change in the foreseeable future. All of the votes and reviews would be relevant even if they are over a year old.

Maybe tie the archive to 90 days prior to the current release upload date or something like that. It wouldn't be particularly hard to code it that way and it would make the system highly accurate to the currently available code.

[edo888]

Yes, I was actually trying to suggest the same. The process of archiving need to be automatic based on the extension update date and may be other parameters too.

I want to suggest another improvement about the extension rating. Can we change the sorting algorithm, so that it will sort by number of 5s, then 4s, then 3s, etc... instead of the average of ratings? So if there is a just one bad vote it will matter less than other marks.

[nonumber]

We (the JED team) are working on a new ranking algorithm too. That will take age of votes into account automatically.
But before we implement that (which still needs some time and changes to take place) we want to make this first easy step to clean the votes/review history.

So please keep this discussion focused on simply that topic.
Removal of votes/reviews older than 2008.

We will open up topic(s) to discuss the ideas around the new ranking algorithm somewhere in the near future.

[edo888]

In that case, if the age of a vote/review will matter, then this will be just an extra work. I don't mind though.

[NHRADeuce]

I agree, if you are working on an algorithm to weight reviews and votes by age, this seems unnecessary for what appears to be a minimal change in accuracy (+.05%).

However, it is a pretty simple change that can be completed with minimal effort so I also don't see any harm in doing a blanket wipe of pre-2008 records.

[mlipscomb]

Removing votes/reviews based on version is not an option at this time and it's doubtful it ever will be. If a dev knows that their 30 votes are going to be removed when an extension is updated, it would make me not want to update.

However, an automated task to archive votes every quarter or month from 18 months ago is feasible.

The possibility of a new ranking/rating system is still quite a ways out, so it shouldn't be a focus right now until more things are ironed out.

[brian]

Whilst I would prefer reviews for older versions be removed I reluctantly accept (even if i disagree with your reasoning) that it won't happen.

What would make the whole votes/reviews more reliable and significant would be the removal of anonymous voting. It's far too open to abuse and just makes a mess of the system. Personally I would make voting a feature restricted to users who are logged in although I know many others would like to see voting restricted to those who write reviews.

[NHRADeuce]

It really wouldn't be hard to make the cut off version upload date minus X days. That keeps votes and reviews from being wiped out completely while still keeping them somewhat accurate and timely. It would be equally enticing for a developer to update sooner to remove a bunch of bad reviews/votes. Keeping X days worth of reviews/votes from the update makes it harder to game the system.

[masterchief]

What an interesting topic. First, thanks for asking.

I don't think reviews should ever be removed/archived. They are a good historical record of the journey that the extension has taken, and how the developer may, or may not, be performing.

On the issue of ratings, I know I've talked to a few people that are frustrated by them and they are certainly open to abuse. Not only that, but there is no real value in it because a 5 from one person could be a 4 from another and so on - in other words, it's completely subjective. Now, a problem I see with archiving votes is that your user suddenly needs to be aware that their votes has expired. That doesn't really make sense because you aren't actually archiving something the developer owns, but something that other people own (the individual votes).

So I was thinking about how that could work better and another site popped into my head: http://osx.iusethis.com/

Maybe the JED can take a look at the methodology here to solve the inherent shortcomings of the 5 star rating system. You have two simple metrics: I use this, and I love this (or this is my favourite). The JED already has favourites. If you implemented a simple "I use this", that actually, to me, carries more weight (possibly) than whether I think this is a 3 or 4 or a 5 star product.

It also solves the problem of a developer self rating because they can rightly use and favourite their own product without any problem.

Now couple that with the ability to sort the category list by:
a) how many people use this extension
b) how many people favour this extension
c) date last updated

That gives me 3 choices to analyse how I'm going to sift through a category but most importantly it moves your 1-5 star subjective rating to a boolean value (I use it and it's one of my favourites or it isn't). The second thing it deals with is the inherent negativity of a 1 or 2 out of 5 vote, and how that can be used against developers (whether fair or not). Any if you then had a page like http://osx.iusethis.com/user/eddieajau, the game becomes very interesting because if so-and-so uses "this" and it's their favourite, and I trust so-and-so's judgement, maybe I should take a look (conversely, if someone has picked everything on the JED, you know they are pulling the wool over your eyes and just being stupid). I think it's a real game changer.

But anyway, if that's going off topic, splice this reply out to a different thread. But I think it's worth thinking about in the context of what to do with the rating problem.

[brian]

Definitely a good idea Andrew and it builds on some ideas that others have discussed previously but it probably is a discussion for a different thread.

[tuum]

vdrover - I agree that old reviews are less relevant as times goes on
masterchief - I agree that the old reviews are a good historical record of how the extension has evolved.

Is it possible to keep the reviews for historical reference, but just not count the votes towards the overall rating?


NHRADeuce - Archiving based on last update would be easy to abuse, anyone with bad reviews could just release frequent minor updates.


brian - I am one of those people who would like to see voting restricted to those who write reviews. The reviews help the developer to know what the users are thinking and what they can do to improve their product. It also gives the developer an opportunity to state their case and correct any missconceptions. Ratings without reviews are a waste of time and too easy to abuse - if someone like or hates something I think they should justify why they gave it a particular score.

It would be great if reviewer had the ability to respond after developer, as based on the developers comments they may rethink their review or clairfy a point. Obviously the developer should have the final closing statement.

[sourcecoast]

I agree that not removing old reviews is a good idea. It is good to see reviews that are as old as the extension, though I'd love it if reviews over a year old were highlighted more as being very old. This could be the date in bold red, or a note before the actual text, etc.

Staying more on-topic regarding voting in particular, I definitely agree with archiving votes from over (almost) 2 years ago. Any extensions votes from then (even if the extension hasn't been updated) would be greatly different now and would fall into one of the below categories:
* If it hasn't been updated in over 2 years, there's probably something better to supplant it, meaning that 2 year old votes may be artificially high and the same user now would vote something else higher.
* If it has been updated recently, then those votes should count more than ridiculously old votes since it could have been greatly improved, or totally broken in that time.

Ideally, I'd like the voting system to only show the average of votes from the trailing year (though I'm open to just about any realistic software-development timeframe). For those that hate the idea of archiving votes, I'd suggest displaying a dual vote, one for all time and one for the last year (for example). This would actually add value as, although voting is arbitrary as Andrew mentioned above, it could actually show a trend of a superior extension (if overall and trailing year were both high), improvements (overall low and trailing higher) and flailing quality (overall is high but trailing is lower).

Also, I'd think, the dual vote report would be easy to implement with the data available, which is really what's important

Finally, votes without reviews suck for a developer. It's amazing that most 5 star votes for our stuff has an attached review, and many lower star votes don't. That makes the reviews almost useless for us as we get no value directly (though the praise makes us sleep better at night). I know it would never happen, but I'd love a system where anything less than 5 stars has to write a review. If you give 5 stars, we understand everything's great, review optional... if you don't like the extension and want to critique us, please do, with a review.

[directtech]

I definitely think this is a good idea. Reviews/ratings from over 2 years ago are completely irrelevant. Actually, anything over 1 year is probably irrelevant... at least for extensions that are actually active. Keeping that in mind, a 2 year mark might make more sense I suppose. Either way, a step in the right direction.

I also agree with Brian that votes without a review should NOT be allowed. It's an easy way for someone to anonymously vote on their own extensions or for competitors to vote against your extensions. A vote should be accompanied by a valid review.

Good stuff

[directtech]

On another note... it would be helpful if the reviews showed the version of the extension at the time of that review. That would be a nice enhancement in this whole review changing proposal. I know this has been brought up before... just throwing it in here again.

[brian]

On another note... it would be helpful if the reviews showed the version of the extension at the time of that review.

I started to write that in my previous post and then I realised that just because a review was written when 1.04 is the current version it doesn't mean that the reviewer was even using that version.

[directtech]

Excellent point Brian... maybe that wasn't such a good idea

[vdrover]

Great comments folks. Lets also try to stay on topic. We'll be discussing more changes to the JED soon.

@masterchief: I'll leave your suggestions in this thread. We'll have a chance to revisit at a later time.

I do like the idea of leaving the older reviews visible (maybe a link to the "archived reviews"?) but removing the old votes from the ratings calculations.

Lets keep the comments coming!

[masterchief]

@masterchief: I'll leave your suggestions in this thread. We'll have a chance to revisit at a later time.

No probs at all.

Just one thing though. Are the listings sorted by the rating alone in descending order or are there other metrics involved as well?

[sourcecoast]

What's the expectation for archived votes? Would the same user be allowed to vote again if their vote were archived, would their votes just be trapped in time, etc?

I honestly don't have any suggestions, just trying to think through possible consequences for extensions older than 2008 from the users that either love, or hate, them still. We don't have any extensions in this timeframe, but we'd be in this net on the next round of archiving.

[mlipscomb]

These are all some great ideas.

I specifically like the idea of "I use this"....

I also really like the idea of adding in other star ratings such as:
x of x stars from the past 6 months.
x of x stars from the past year.
x of x stars from the past 2 years.
And then of course the overall rating.

This would provide the ability to see the "rating history" of the listing and it's something that is pretty easy to implement.

What's the expectation for archived votes? Would the same user be allowed to vote again if their vote were archived, would their votes just be trapped in time, etc?

Great question - will find out the answer. This is not something that has been done before, so I'm sure there will be many questions/answers before a final decision is made.

I definitely believe we are on the right path to making the voting/reviews more relevant and fair to all.

[Oceanwatcher]

First of all, I would like to have reviews available for ever. It shows the development of both the extension and the people behind the extension.

The problem with cutting off votes older than a certain time is that we have no idea what version a vote was based on. But I would love to see something like what WordPress is using. Check this one:

http://wordpress.org/extend/plugins/sociable/

On the right side is a simple Broken/Works vote. And in that module, you can also choose the version of WordPress and the version of the extension. So here you can check the version you are planning to install and see if there is any chance it will work.

This would help making votes relevant. I don't think the stars are that useful at all. The content in the reviews are the most important.

So I would prefer to see a system where only those writing a review can say if it works or not and they have to choose the version they are reviewing.

[brian]

As voting is currently anonymous there is nothing to stop a user voting every day if they wanted to. So archiving votes would have no effect on limiting a users rights to vite

[nonumber]

Hard to stay on topic, hey guys.

I know you all want to discuss the voting system. And we (JED team) have been discussing it endlessly too. As said a few times in this thread, we will open up discussion on voting/rating system soon.

The focus and question now is simply:
Do you oppose to us removing (archiving) the votes/reviews from before 2008.
And if so, why?

There is a lot of junk activity in the before-2008-period.
It would help us greatly in our efforts to keep the JED clean and fair to simply remove (archive) that activity.
Seeing the effect is pretty minimal, I don't see a problem with that.
Extensions shouldn't be reliant on votes/reviews from more than 2 years ago.
Also having a history log on reviews might sound nice, but who actually goes through reviews from more than a few months old?! I think virtually no-one.

[edo888]

The voting is not anonymous Brian, at least it is visible to the JED Team, when you click to vote without logging in it throws error.

I agree with Andrew, that votes/reviews shouldn't be removed at all. It will be better to leave it as is and at least don't count them as suggested earlier. And it will be really positive to have a boolean value instead of 1-5 rating. It will keep votes positive always.

[edo888]

Can we ask every developer individually if s/he wants his extension reviews be archived or not?

[vdrover]

Can we ask every developer individually if s/he wants his extension reviews be archived or not?

This is not practical i'm afraid.

[edo888]

Why not? I just found a extension ID which is published on 3 Jan 2008. http://extensions.joomla.org/extensions/3850 . I guess there are about 1000 extensions and about 300 developers. So only that 300 are going to loose their extension reviews (even less, since not all those extensions have reviews before 1 Jan 2008). You can create a web form hosted somewhere and email the devs and give them let's say 10 days to respond. Who says that he don't want, his ext. reviews will not be removed. After that you will possible run 1 query to remove reviews.

If you need help for implementation I'm ready to assist.

[Oceanwatcher]

There is a lot of junk activity in the before-2008-period.
It would help us greatly in our efforts to keep the JED clean and fair to simply remove (archive) that activity.
Seeing the effect is pretty minimal, I don't see a problem with that.
Extensions shouldn't be reliant on votes/reviews from more than 2 years ago.
Also having a history log on reviews might sound nice, but who actually goes through reviews from more than a few months old?! I think virtually no-one.

Step one: Cut everything that is older then one year before last update of the extension.

Step two: Cut the whole voting system as it is today!

For me, one of the reasons for an action like you are proposing would be to be able to get a listing of ext that is more relevant. Not alphabetically and not purely based on voting. If this is the result, I am all for cutting. Cut to the bone! :-)

About going back in time on reviews: I do. I go far back. I am actually always looking for the bad reviews. The excellent ones are not too relevant to me. To read 20 reviews of "Excellent product, excellent support" is not as interesting as to read a well founded negative report and see how the dev. respond to it. This tells a lot more about the ext. and the dev. But as you said - this is beside the point for this discussion :-)