Page 1 of 1

JoomlaCode is closing

Posted: Fri Jan 30, 2015 7:11 pm
by ChiefGoFor
Please be aware that JoomlaCode will coming down "in the very near future". For more details, please read the official statement on the Joomla! Developer Network page:

http://developer.joomla.org/news/602-jo ... osing.html

Re: JoomlaCode is closing

Posted: Tue Feb 03, 2015 9:24 pm
by H13
Hi to all.

Please, is there someone who can give me some clue advice or help how to move over 1000 Joomla! files (extensions, translations) from joomlacode.org to another server (not manually).


Second question? What about Joomla! project itself. It is planned to continue with Joomla! project? Should I migrate my extensions to other CMSs like Wordpress or Drupal? I mean, closing Joomlacode.org is really fatal decision. I understand, I fight spam attacks and DDOS attacks since 2007, so it is really uncomfortable situation. But it is really needed to stop the project?

:-(

Thank you, Jan

Re: JoomlaCode is closing

Posted: Tue Feb 03, 2015 9:28 pm
by mbabker
There are some API references in the linked article which can help you to back up any data from JoomlaCode.

The Joomla project itself isn't going anywhere. Quite simply, the project can no longer to maintain JoomlaCode as a service for a variety of reasons and this specific service is being shut down. It affects the project as well since Joomla has used it to store all of its downloads and has the 1.0 and 1.5 code repositories, in addition to translation resources. So we're going through the same pains to move our resources, which are exponentially larger than most projects on JoomlaCode.

Re: JoomlaCode is closing

Posted: Tue Feb 03, 2015 9:35 pm
by H13
Hi Michael,

thank you very much for your info.

Are there any recommendations from you or Joomla! project, where to migrate. Is GitHub the one preferred?

I am planning to add automatic update for some of my extensions in the future (planned for joomlacode.org files). Will there be no problems on other services?

EDIT:
Is there any date when it will be closed?

Thank you, Jan

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 9:29 am
by piotr_cz
I can recommend GitHub for hosting free Joomla extensions.
Issues/ Pull requests/ Wiki/ Releases/ Pages, all one can possibly need.

It's quite comfortable to use as a update server too, you just make a tag with new version, create a release and add download link to xml file (Check extension on my profile https://github.com/piotr-cz or look for extensions hosted on GitHub, there are few around).

Only one disadvantage I found is that you will get very little analytical data (just basic traffic info) and no data of downloads/ updates but you can always host update server yourself.

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 9:34 am
by brian

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 11:56 am
by Beat
When reading above posts from Joomla leaders, which convey positive great news, but are begining with:

"JoomlaCode will coming down", "The Joomla project itself isn't going anywhere."

Wow!!! Doesn't that read a bit short as intros ?

Those statements are imho scary as is in here (and maybe elsewhere too?) for diagonal readers and could be read both ways (and as they are scary, odds are they get read in the wrong, negative, way, like happened above).

Similarity in name of "Joomlacode" (.org) and "Joomla code" (".php") is certainly the confusion that triggered the 2nd part of this thread's first reply.

Probably an edited more positive introduction of those posts could do only clarify the positive message and do good to the project.

Proposals (constructively):

"While the Joomla code has moved to Github since May 2013, the old forge site "joomlacode.org" will finally be closing down."

"The Joomla code is continuing to go forward just fine on github, where it moved since May 2013 and wirh regular new releases."

That is *not* critics, my English isn't great either and sometimes I write some miscommunication gems that can be read both ways (i'm sure this could happen to this post too, even i read it 2 times, but other eyes will read it differently for sure), just a different reading trying to understand why the message didn't come through positively in the repyl and even misunderstood right there.

Positively,

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 4:04 pm
by piotr_cz
Thanks, this looks really helpful. Unfortunately I can't see the key download_count in the response, but I'll investigate.

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 4:17 pm
by brian
Not used it - just googled to find it

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 5:17 pm
by H13
Hi,

hmm, I discovered, that the 1000 files I have posted to joomlacode.org are only the "active" files - not all the tree of obsolete versions (which are asked by many users too).

So the questions:

Which way will Joomla! core files migrated? To which server?

I have tried to run somehow this:
https://github.com/mbabker/bug-squad-st ... gforge.php

but without any success.

Are there any more detailed information? I think, on joomlacode.org there is such large source of useful files that I will be not the only one who needs to solve this issue.

If anybody plans to do some helpful scripts, please let me know - I can participate on this (developing or testing).

Thank you, Jan

Beat
You fully right, "JoomlaCode is closing" is really not good PR message for the project and it both cases:
1) as the header of article
2) as the fact that some core service will be moved outside - e.g. instead evolving it with "extensions market" services, etc. etc.

:-( :-(

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 8:03 pm
by mbabker
I'm working on some of the project for moving Joomla's own data off JoomlaCode and to new systems (I'll let others explain that in more detail when the time is right). The GForge connector classes we linked in the announcement were in use on developer.joomla.org to extract much of our issue tracker data to the site and enabled us to create the charts linked from http://developer.joomla.org/tracker/vol ... ivity.html or the release note pages that are linked from http://developer.joomla.org/tracker.html. Those classes don't have all of the available API methods that GForge offers (if you review the links to those shared in the announcement). They really make it easy to get going with though.

If you're using those classes, this snippet is how you instantiate them (same applies for the legacy class):

Code: Select all

$gforge = new GForge('http://joomlacode.org/gf');
$gforge->login('username', 'password');
Once you've logged in, you should be able to use the other methods in the GForge class. It's really just a wrapper around PHP's SoapClient class for interfacing with a SOAP API, there isn't much special about it.

Another use case you're probably interested in is exporting your packages to a new hosting platform. You can use https://gist.github.com/mbabker/24ab73694fdd3b95fdc8 as a starting point for this; this is a command line script that can download all of the files for an array of release IDs (as that's how they're grouped in JoomlaCode). This sample will fetch all of the Joomla 1.0 downloads.

Re: JoomlaCode is closing

Posted: Wed Feb 04, 2015 11:26 pm
by H13
Hi Michael,

thank you for the info.

I have a problem with SOAP5 - http://joomlacode.org/gf/xmlcompatibility/soap5/#

Everything seems to be easy until I need the files:

1) I can login with gforge class
2) I can get info about all my projects
3) about all my packages
4) about all my releases

I didn't find any function for getting packages and releases in the gforge.php class, so I have added them:

Code: Select all

- $this->client->getFrsPackages($this->sessionhash, $projectId);
- $this->client->getFrsReleases($this->sessionhash, $packageId, $userId);
No problem, but now I have all the info and need to get information about the files.

in SOAP I see the method: getFiles, but SOAP does not work, only SOAP5.

In your class ( https://gist.github.com/mbabker/24ab73694fdd3b95fdc8 ), I see, you call the following method:

Code: Select all

$gforge->getFilesystemsForRelease
which is not a part of gforge.php library: https://github.com/mbabker/bug-squad-st ... gforge.php I have downloaded.

So the question is, which SOAP5 method you are using for getting info about the files stored in package/release?


Thank you for your time.

Jan

Re: JoomlaCode is closing

Posted: Thu Feb 05, 2015 12:50 am
by mbabker
The updated GForge class I'm working with is at https://gist.github.com/mbabker/044811b5279daa5c7f48. There are 6 methods without doc blocks near the bottom of the file I was working with yesterday to make a few updates and pull down the data I needed.

Re: JoomlaCode is closing

Posted: Fri Feb 06, 2015 11:51 pm
by H13
Hi, thank you very much:

Projects - count: 23
Packages - count: 103
Releases - count: 1,383
Files - count: 1,468

OK, there are three steps:

1) Get information about files on joomlacode.org - build database structure - download them all and build folder/file tree - DONE
2) Find another server and publish the projects/packages/releases/files there (automatically :idea: ) NOT DONE
3) Rebuild the links to new server NOT DONE

Michael, do you have some tips for 2) or 3) ? Or does anyone has tip for both issues?


If anyone is interested how to do 1), I did it this way (the script is not fully OOP and I didn't care about security, just made a quick solution for only one use - quickly get the info about the projects/packages/releases/files):

a) downloaded gforge.php provided by Michael (thank you very much) - https://gist.github.com/mbabker/044811b5279daa5c7f48

b) extended:

Code: Select all

public function getPackagesByProject($projectId) {
        try {
            $a = $this->client->getFrsPackages($this->sessionhash, $projectId);
            return $a;
        } catch (SoapFault $e) {
            throw new RuntimeException('Error : ' . $e->faultstring);
        }
    }
    
    
    public function getReleasesByPackage($packageId, $userId) {
        try {
            $a = $this->client->getFrsReleases($this->sessionhash, $packageId, $userId);
            return $a;
        } catch (SoapFault $e) {
            throw new RuntimeException('Error : ' . $e->faultstring);
        }
    } 
c) in next post ...

Re: JoomlaCode is closing

Posted: Sat Feb 07, 2015 12:02 am
by H13
c) I didn't run php in command line and I have a lot of files on joomlacode.org so I have used ajax (async: false which is not recommended but for my needs OK), see the main parts of the code:

Code: Select all

JHtml::_('jquery.framework', false);
$doc        = JFactory::getDocument();
$db         = JFactory::getDBO();
$gf         = new GForge('http://joomlacode.org/gf');
$gf->login($userName, $password);
$bU         = 'http://joomlacode.org';
$user         = $gf->getUser();
$projects     = $gf->getUserProjects();

$i            = 0;
$jsAllRel    = array();

echo '<div class="alert alert-info" id="info" style="float:right;"></div>';
echo '<div style="clear:both"></div>';

if (!empty($projects)) {
    foreach($projects as $k => $v) {
        echo '<h1 style="background: #f0f0f0;border:1px solid #ccc;padding: 5px;">'.$v->project_name.'</h1>';

        $query = ' INSERT INTO #__phocautils_projects (project_id, project_name, unix_name)'
                .' VALUES ('.(int)$v->project_id.', '.$db->quote($v->project_name).', '.$db->quote($v->unix_name).')'
                .' ON DUPLICATE KEY UPDATE project_name = VALUES(project_name),'
                .' unix_name = VALUES(unix_name)';
        $db->setQuery($query);
        $db->execute();
        
        $packages     = $gf->getPackagesByProject($v->project_id);
        if (!empty($packages)) {

            foreach ($packages as $k2 => $v2) {
                echo '<h2>'.$v2->package_name.' ('.$v2->frs_package_id.')</h2>';
                echo '<div><a href="javascr...ipt:void(0)" on...click="getReleasesAndFiles('.$v2->frs_package_id.', '.$i.')">Get Releases and Files</a></div>';
                echo '<div id="result'.$i.'"></div>';
                $jsAllRel[] = $v2->frs_package_id;
                
                $query = ' INSERT INTO #__phocautils_packages (frs_package_id, project_id, package_name)'
                .' VALUES ('.(int)$v2->frs_package_id.', '.(int)$v2->project_id.', '.$db->quote($v2->package_name).')'
                .' ON DUPLICATE KEY UPDATE project_id = VALUES(project_id),'
                .' package_name = VALUES(package_name)';
                $db->setQuery($query);
                $db->execute();
                
                $i++;
                
                // Because of time issues - will be solved by AJAX
                /*$releases     = $gf->getReleasesByPackage($v2->frs_package_id, $user->user_id );
                if (!empty($releases)) {
                    foreach ($releases as $k3 => $v3) {
                        $files        = $gf->getFilesystemsForRelease($v3->frs_release_id);
                        if (!empty($files)) {
                            foreach ($files as $k4 => $v4) {
                                echo '<div>'.$v4->download_url.'</div>'. "\n";
                            }
                        }
                    }
                }*/
            }
        }
    }
}

if (!empty($jsAllRel)) {
    echo '<div class="btn" style="float:right;" on...click="getAllReleasesAndFiles();">Import all releases and files at once</div>';
}

$url     = 'index.php?option=com_phocautils&view=import&format=json&tmpl=component&'. JSession::getFormToken().'=1';
$s         = array();
if (!empty($jsAllRel)) {
    $s[]    = ' function getAllReleasesAndFiles() {';
    $s[]    = '   var a =['.implode(',', $jsAllRel).'];';
    $s[]    = '   for (i = 0; i < a.length; ++i) {';
    $s[]    = '     getReleasesAndFiles(a[i], i);';
    $s[]    = '     jQuery(info).html(\'Importing of release \' + a[i] + \' done\');';
    $s[]    = '   }';
    $s[]    = ' }';
}
$s[]    = ' function getReleasesAndFiles(packageId, id) {';
$s[]    = '   var result = \'#result\' + id';
$s[]     = '   jQuery.ajax({';
$s[]     = '      url: \''. $url.'&id=\' + packageId + \'\',';
$s[]     = '      type:\'GET\',';
$s[]     = '      data:\'\',';
$s[]     = '      dataType:\'JSON\',';
$s[]    = '         async: false,';
$s[]     = '      success:function(data){';
$s[]     = '         if ( data.status == 1 ){ ';
$s[]     = '            jQuery(result).html(data.message);';
$s[]     = '         } else { ';
$s[]     = '               jQuery(result).html(data.error);';
$s[]     = '         }';
$s[]     = '      }';
$s[]     = '   });';
$s[]    = ' }';
$doc->addScriptDeclaration(implode("\n", $s));
and the ajax code:

Code: Select all

if (!JRequest::checkToken('request')) {
            $response = array(
                'status' => '0',
                'error' => '<div class="alert alert-error">' . JText::_('JINVALID_TOKEN') . '</div>');
            echo json_encode($response);
            return;
        }
        
        $app        = JFactory::getApplication();
        $db         = JFactory::getDBO();
        $id            = $app->input->get( 'id', '', 'int'  );
        
        $gf         = new GForge('http://joomlacode.org/gf');
        $gf->login($userName, $password);
        $bU         = 'http://joomlacode.org';
        $user         = $gf->getUser();
        
        $releases     = $gf->getReleasesByPackage($id, $user->user_id );
        $o = '';
        $i = $j = 0;
        if (!empty($releases)) {
            foreach ($releases as $k3 => $v3) {
                $i++;
                $query = ' INSERT INTO #__phocautils_releases (frs_release_id, frs_package_id, release_name, release_date)'
                .' VALUES ('.(int)$v3->frs_release_id.', '.(int)$v3->frs_package_id.', '.$db->quote($v3->release_name).', '.$db->quote($v3->release_date).')'
                .' ON DUPLICATE KEY UPDATE frs_package_id = VALUES(frs_package_id),'
                .' release_name = VALUES(release_name),'
                .' release_date = VALUES(release_date)';
                $db->setQuery($query);
                $db->execute();
            
                $files        = $gf->getFilesystemsForRelease($v3->frs_release_id);
                if (!empty($files)) {
                    foreach ($files as $k4 => $v4) {
                        $j++;
                        $query = ' INSERT INTO #__phocautils_files (filesystem_id, frs_release_id, frs_package_id, file_name, file_name_safe, file_size, download_count, download_url)'
                        .' VALUES ('.(int)$v4->filesystem_id.', '.(int)$v3->frs_release_id.', '.(int)$v3->frs_package_id.', '.$db->quote($v4->file_name).', '.$db->quote($v4->file_name_safe).', '.(int)$v4->file_size.', '.(int)$v4->download_count.', '.$db->quote($v4->download_url).')'
                        .' ON DUPLICATE KEY UPDATE frs_release_id = VALUES(frs_release_id),'
                        .' frs_package_id = VALUES(frs_package_id),'
                        .' file_name = VALUES(file_name),'
                        .' file_name_safe = VALUES(file_name_safe),'
                        .' file_size = VALUES(file_size),'
                        .' download_count = VALUES(download_count),'
                        .' download_url = VALUES(download_url)';
                        $db->setQuery($query);
                        $db->execute();
                        
                    }
                }
            }
        }
        $o .= '<div>Number of releases: '.$i.'</div>';
        $o .= '<div>Number of files: '.$j.'</div>';
        
        $response = array(
        'status'    => '1',
        'message' => '<div class="alert alert-success">' . $o . '</div>');    
        echo json_encode($response);
        return;
change: on...click and javascri...pt to right names (added ... because of security reasons in this forum, it is not possible to add such strings)

Jan

Re: JoomlaCode is closing

Posted: Sat Feb 07, 2015 12:27 am
by brian
Thanks for sharing your code I hope it will be useful for others

Re: JoomlaCode is closing

Posted: Sat Feb 07, 2015 1:16 am
by mbabker
For our case, we're building out an Akeeba Release System implementation and integrating with Amazon S3 for file storage, so our complete script (the snippet I shared is just downloading all the packages to the local system) is downloading the package from JoomlaCode, uploading it immediately to the S3 bucket, then removing it from the local system. As we build out everything, the package mapping will take care of itself.

As for your items 2 and 3 from above, those are generally going to be unique to each project who needs to back up their data. I think the pain points are going to be in backing up any data from JoomlaCode, and there is an API to do it and a small set of tools the project has been using for other resources (all of which we shared in the initial blog) to help make this possible. In my case, I have an extension listed on JoomlaCode but the only remaining connection there is the old release packages, and considering those versions will not work anymore (and I've unlisted the downloads from my own site), I'll probably do nothing more than get the file count for each package and record that data.

Our other major task is handling redirects for links to the packages. Generally, we're building redirects for each individual package and the package listing page (the one that lists all of the files in a release). This is going to end up being a rather large list (just for the 3.x releases, there are 288 individual packages) and one that's going to need a lot of time to prepare and review.

Re: JoomlaCode is closing

Posted: Wed Feb 11, 2015 4:38 pm
by xillibit
I'am wondering if there is a way to export statistics from joomlacode project because it can be insteresting to don't lose them but i have looked but there are nothing it seem to handle statistics export or maybe i'am wrong ?

Re: JoomlaCode is closing

Posted: Wed Feb 11, 2015 8:16 pm
by mbabker
Which statistics are you looking for?

Re: JoomlaCode is closing

Posted: Wed Feb 11, 2015 8:20 pm
by xillibit
The statistics about number of packages downloaded that you can see under "Reporting" and then "Download report"

Re: JoomlaCode is closing

Posted: Thu Feb 12, 2015 6:22 pm
by mbabker
Using the SOAP API's getFilesystems method, you can get an array of filesystem data objects. The method declaration is getFilesystems($sessionhash, 'frsrelease', $releaseId) where $sessionhash is the hash you receive after authenticating to the API and $releaseId is the release ID you want to retrieve the files for.

In each of those objects is a "download_count" key which has the total download count. You'll need the release ID for each item in question. Using Joomla's release listing as an example, from http://joomlacode.org/gf/project/joomla/frs/ you can click on each release (the second table column) to get its release ID, and that is what you use when calling this API endpoint.

Re: JoomlaCode is closing

Posted: Sun Apr 19, 2015 5:15 pm
by awair
New to Joomla and the Forum.

Can anyone let me know which template joomlacode.org is using and whether it is freely available? Or who I should contact?

Many thanks.

Re: JoomlaCode is closing

Posted: Sun Apr 19, 2015 5:54 pm
by JoomIa-Shark
So no more SigPlus???

Re: JoomlaCode is closing

Posted: Fri May 01, 2015 6:58 pm
by xillibit
mbabker wrote:Using the SOAP API's getFilesystems method, you can get an array of filesystem data objects. The method declaration is getFilesystems($sessionhash, 'frsrelease', $releaseId) where $sessionhash is the hash you receive after authenticating to the API and $releaseId is the release ID you want to retrieve the files for.

In each of those objects is a "download_count" key which has the total download count. You'll need the release ID for each item in question. Using Joomla's release listing as an example, from http://joomlacode.org/gf/project/joomla/frs/ you can click on each release (the second table column) to get its release ID, and that is what you use when calling this API endpoint.
I have tried that and i'am able get the data, with you cool gforge class

On joomlacode in the project, if you go under Reporting -> Downloads Report, you can choose a period to show downlaod stats in this period. But the download stats are stored by dates somewhere ?

Re: JoomlaCode is closing

Posted: Fri May 01, 2015 11:19 pm
by mbabker
Sadly that data isn't API accessible it seems. Looking at a copy of the GForge software I managed to get my hands on, that entire page is built by direct SQL queries and rendering each row of the results.

The full API is somewhat documented at http://joomlacode.org/gf/xmlcompatibility/soap5. Maybe you can find a method in there to get the data you're looking for, but at a quick glance I don't see an endpoint returning download records.

Re: JoomlaCode is closing

Posted: Wed Dec 23, 2015 2:45 pm
by jojo7
Hi,

looks like some people (new Joomla! users?) still download from http://joomlacode.org/gf/project/joomla/frs/ (download-counter goes up...)

maybe no nice user experience when they realise version is vulnerable

I already posted in wrong board, sorry: http://forum.joomla.org/viewtopic.php?f=7&p=3351735

best regards

Re: JoomlaCode is closing

Posted: Wed Dec 23, 2015 2:48 pm
by ChiefGoFor
Thank you Jojo... I will make sure the right people are notified and this is taken care of. Thank you for bringing this to our attention. :)

Re: JoomlaCode is closing

Posted: Wed Dec 23, 2015 4:39 pm
by ChiefGoFor
Hey Jojo. Thank you for bring this to our attention. Robert Deutz has taken care of this.

Re: JoomlaCode is closing

Posted: Thu Dec 24, 2015 11:22 pm
by jojo7
Great, thank you!